0 


AD-A258  912 


AFIT/GSE/ENY/92D-03 


DTIC 


ELECTE 
JAN  7  1993 


I 


Integrated  Blade  Inspection  System  Upgrade  Study 


THESIS 

GSE-92D 

AFIT/GSE/ENY/92D-03 


o/w§3_00030 

IliHIDIII  Tri) 


Approved  for  public  release;  distribution  unlimited 


93  1  04  039 


AFIT/GSE/ENY/92D-03 


Integrated  Blade  Inspection  System  Upgrade  Study 


THESIS 


Presented  to  the  Faculty  of  the  School  of  Engineering 
of  the  Air  Force  Institute  of  Technology 
Air  University 
In  Partial  Fulfillment  of  the 
Requirements  for  the  Degree  of 
Master  of  Science  in  Systems  Engineering 


Tony  J.  DeLiberato,  B.S.,  B.A. 
First  Lieutenant,  USAF 


Steven  W.  Perkins,  B.S. 
Captain,  USA 


Steven  K.  Saplin,  B.S. 
Captain,  USAF 


John  G.  Snyder,  B.S. 
Captain,  USAF 


Gregory  J.  ToussaJnt,  B.S. 
First  Lieutenant,  USAF 


December,  1992 


DTIC  QUALH  '. 


Ac«««slcm  For 
NTI5  QMAkl 

Ju  .  '  t  i  f  t.  i  otL 


Br _ 

D  i  r' t  r  1  \  •>:,'!/ 

Av.«  ‘  Coc 

'Dlst  : 


Approved  for  public  release;  distribution  unlimited 


son 


Preface 


The  purpose  of  this  design  study  was  to  identify  ways  to  improve  the  performance 
of  the  Integrated  Blade  Inspection  System  (IBIS).  The  current  system  uses  an  automated 
visual  inspection  system  in  conjunction  with  human  inspectors  to  identify  cracked  jet  engine 
blades  that  have  been  processed  with  fluorescent  penetrant.  The  system’s  major  problems 
lie  in  the  automated  inspection  system  which  is  difficult  and  expensive  to  maintain,  and 
unable  to  correctly  classify  cracked  blades. 

We  applied  a  systems  engineering  design  process  to  evaluate  the  current  inspection 
techniques  and  to  develop  alternative  methods  to  satisfy  the  Air  Force  requirements.  We 
identified  deficiencies  in  both  the  manual  and  automated  inspection  processes.  Based  on 
these  deficiencies,  we  posed  three  different  inspection  systems  to  compare  to  the  current 
process.  The  first  alternative  relied  on  manual  inspection  for  all  of  the  blades.  The  second 
option  utilized  a  semi -automated  process  with  both  automatic  and  manual  inspection.  The 
third  approach  was  a  fully  automated  blade  inspection  system  that  required  no  human 
inspectors.  Based  on  our  research,  it  seems  that  the  fully  automated  system  holds  the 
most  promise  in  addressing  the  Air  Force’s  needs. 

We  would  like  to  thank  all  the  people  who  selflessly  provided  their  time  and  support 
to  make  this  study  possible.  Specifically,  we  would  like  to  thank:  our  faculty  advisors. 
Dr  Curtis  Spenny  and  Maj  David  Robinson  for  their  advice  and  support;  our  commit¬ 
tee  members  Maj  Steven  Rogers,  and  Maj  Mike  Leahy  for  their  diligence  and  insight; 
our  sponsor  SA-ALC/TIEST  for  the  opportunity  to  solve  a  real-world  problem;  Mr  Ward 
Rummel  of  Martin  Marietta  for  his  advice  and  support;  the  General  Electric  Quality  Tech¬ 
nology  Division  for  their  willingness  to  share  background  information  and  provide  insights 
in  the  nondestructive  evaluation  arena;  Wright  Laboratory  Materials  Directorate  for  the 
use  of  their  facilities  and  expertise  in  fluorescent  penetrant  inspection;  the  mafly  vendors 
who  answered  our  never-ending  questions;  and  especially  our  families  who  encouraged  and 
supported  us  throughout  the  study. 
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Abstract 

The  purpose  of  this  design  study  was  to  identify  ways  to  improve  the  performance  of 
the  Integrated  Blade  Inspection  System  (IBIS).  The  Air  Force  requires  routine  inspection  of 
jet  engine  compressor  and  turbine  blades  to  locate  defects  and  prevent  catastrophic  engine 
failure.  The  current  inspection  process  uses  fluorescent  penetrant  as  an  aid  to  identify 
cracked  blades.  Human  inspectors  examine  the  majority  of  the  blades,  but  automated 
inspection  systems,  such  as  the  Fluorescent  Penetrant  Inspection  Module  (FPIM),  are 
also  in  use.  Automated  inspection  potentially  offers  advantages  over  manual  inspection  in 
terms  of  the  size  of  defect  that  can  be  detected,  the  probability  of  detecting  flaws,  and  the 
consistency  of  the  inspection  process. 

A  systems  engineering  design  process  was  applied  to  evaluate  the  current  inspection 
techniques  and  to  develop  alternative  methods  to  satisfy  the  Air  Force  requirements.  The 
study  identified  deficiencies  in  both  the  manual  and  automated  inspection  processes.  Three 
different  inspection  systems  were  developed  and  compared  to  the  current  process.  The  first 
alternative  relied  on  manual  inspection  for  all  of  the  blades.  The  second  option  utilized  a 
semi-automated  process  with  both  automatic  and  manual  inspection.  The  third  approach 
was  a  fully  automated  blade  inspection  system  that  required  no  human  inspectors.  This 
study  made  several  noteworthy  contributions  by: 


•  identifying  the  need  for  and  prescribing  a  statistical  quality  control  plan  for  the 
inspection  process 

•  developing  classification  software  to  validate  the  concept  of  using  neural  networks  to 
classify  blades  more  accurately 

•  demonstrating  the  potential  advantages  of  using  charge-coupled  device  (CCD)  cam¬ 
eras  for  data  gathering 

•  providing  a  means  to  quantify  a  portion  of  the  cost  of  incorrectly  classifying  jet  engine 
blades 

•  identifying  a  method  of  using  multiple  images  to  more  accurately  classify  blades 


The  study  demonstrates  that  the  fully  automated  system  could  dramatically  outper¬ 
form  the  manual  inspection  process  by  improving  the  consistency  of  the  inspection  process 
and  raising  the  quality  of  the  blades  returned  to  service.  All  efforts  should  be  made  to 
pursue  a  fully  automated  inspection  system  because  it  offers  the  best  potential  to  reduce 
the  effects  of  misclassifying  blades. 

In  the  near  term,  the  FPIM  should  be  removed  from  service  because  its  maintenance 
and  operating  costs  are  too  high.  The  manual  inspection  process  should  be  improved 
by  more  closely  monitoring  the  system’s  performance  and  by  applying  statistical  quality 
control  techniques.  Future  plans  should  include  continued  research  to  develop  the  next 
generation  semi-automated  inspection  system,  which  is  viable  with  currently  available 
technology  and  some  additional  development.  After  successful  implementation  and  with 
continual  improvement  of  a  semi-automated  system,  a  fully  automated  inspection  system 
should  follow  as  a  natural  evolution  rather  than  a  complete  redesign. 


Integrated  Blade  Inspection  System  Upgrade  Study 


1.  Introduction 


1.1  Background  Information 

The  Air  Force  spends  tens  of  millions  of  dollars  each  year  performing  unscheduled 
maintenance  on  jet  aircraft  engines  (62,  63).  These  unscheduled  maintenance  events  not 
only  affect  the  Air  Force  in  monetary  terms,  but  also  in  terms  of  aircraft  readiness.  Jet 
engine  blades  that  have  significantly  large  cracks  contribute  to  the  cause  of  a  portion  of  the 
unscheduled  maintenance  events.  Cracked  blades  in  an  engine  can  cause  decreased  perfor¬ 
mance  leading  to  an  unscheduled  maintenance  event  or  catastrophic  engine  failure  which 
could  result  in  the  loss  of  an  aircraft  and  possibly  the  loss  of  life.  Preventive  maintenance 
to  preclude  the  effects  of  cracked  blades  in  jet  engines  includes  periodic  inspection  of  jet  en¬ 
gine  components  to  ensure  reliable  performance  and  to  prevent  accidents  caused  by  engine 
failure.  Compressor  and  turbine  blades  are  two  critical  engine  parts  that  can  be  damaged 
due  to  the  high  stresses  placed  on  the  blades  during  normal  operation.  The  blades  must 
be  periodically  removed  from  the  engine  and  inspected  for  cracks,  scratches,  dents,  pits, 
blocked  cooling  holes,  and  other  flaws.  Crack  type  defects  are  usually  not  visible  to  the 
unaided  eye,  so  the  blades  are  prepared  with  fluorescent  penetrant  (FP)  to  highlight  cracks 
during  inspection.  Only  a  small  percentage  of  the  blades  are  actually  cracked,  making  the 
inspection  task  tedious.  The  inspectors  must  sort  through  a  large  number  of  non-defective 
blades  to  find  the  few  that  actually  contain  flaws.  Human  inspectors  have  the  proper  skills 
to  identify  cracks,  but,  because  of  human  nature,  they  do  not  always  perform  well.  With 
this  in  mind,  the  Air  Force  has  pursued  automated  blade  preparation  and  inspection  to 
improve  the  consistency  and  accuracy  of  the  overall  inspection  process. 

The  Integrated  Blade  Inspection  System  (IBIS)  performs  automated  FP  processing 
and  semi-automated  inspection  of  engine  blades.  The  system  is  located  at  the  Jet  Engine 
Division  of  San  Antonio  Air  Logistics  Center  (SA-ALC/LP)  at  Kelly  AFB,  Texas,  where 
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F-lOO  engines  are  maintained.  The  management  at  SA-ALC  was  pleased  with  the  initial 
success  of  the  IBIS,  but  after  several  years  of  operation,  they  believe  the  system  can  be 
improved  to  further  enhance  their  inspection  capability. 

The  underlying  problems  of  the  IBIS  are  its  lack  of  flexibility  and  the  high  percentage 
of  false  calls  made  by  the  Fluorescent  Penetrant  Inspection  Module  (FPIM).  The  FPIM 
automatically  inspects  blades  for  FP  indications  and  labels  the  blades  as  good,  cracked, 
or  unreadable.  Blades  labeled  as  good  exit  the  system,  and  the  other  blades  are  treated  as 
suspect  and  reinspected  by  a  human  inspector  to  determine  if  they  actually  contain  flaws. 
A  false  call  consists  of  labeling  a  nondefective  blade  as  suspect,  which  causes  the  human 
operator  to  reinspect  the  blade. 

The  original  FPIM  design  does  not  allow  for  modular  upgrades  or  simple  system  im¬ 
provements.  The  computer  system  and  software  are  intimately  connected,  which  makes  it 
difficult  to  modify  any  single  portion  of  the  system.  Many  FPIM  components  were  devel¬ 
oped  especially  for  the  FPIM  and  are  not  readily  available  which  adds  to  its  upgradability 
and  maintainability  problems. 

In  addition  to  the  problems  with  upgrading  and  maintaining  ti  e  system,  the  inspec¬ 
tion  performance  of  the  FPIM  has  not  met  SA-ALC’s  expectations.  It  has  high  false  call 
and  low  utilization  rates.  The  false  call  rate  exceeds  fifty  percent,  which  places  a  large 
reinspection  burden  on  the  inspector  operating  the  FPIM.  In  its  present  configuration,  the 
FPIM  is  only  capable  of  inspecting  stage  4  and  stage  9  compressor  blades  which  account 
for  approximately  11  percent  of  the  total  blades  that  pass  through  the  IBIS.  Even  though 
the  FPIM  is  capable  of  inspecting  11  percent  of  the  blades,  it  was  only  used  to  inspect  less 
than  5  percent  of  the  total  in  fiscal  year  1991.  To  improve  the  present  automated  system, 
it  must  be  made  cost  effective  and  maintainable,  have  the  potential  to  be  upgraded  if  new 
technology  becomes  available,  process  blades  quickly,  and  cissure  that  quality  parts  are 
returned  to  service. 
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1.2  Systems  Engineering  Study 

The  personnel  at  the  Jet  Engine  Division  in  conjunction  with  the  Air  Force  Materiel 
Center-Robotics  and  Automation  Center  for  Excellence  (AFMC-RACE),  also  located  at 
Kelly  AFB,  have  proposed  a  systems  engineering  study  to  investigate  the  present  inspection 
system.  The  objective  of  the  study  was  to  recommend  changes  to  the  IBIS  that  would 
improve  the  FP  inspection  process  for  jet  engine  blades  at  SA-ALC.  The  major  areas  of 
concern  were  the  quality  of  the  FP  application  process  as  well  as  the  performance  of  the 
FPIM  and  the  human  inspectors. 

The  systems  engineering  study  compared  three  alternative  inspection  systems  to  the 
present  configuration.  The  three  alternative  systems  represent  three  possible  production 
methods  that  could  be  used  to  inspect  jet  engine  blades:  manual,  semi-automated,  and  au¬ 
tomated.  The  first  option  is  an  entirely  manual  inspection  system  that  removed  the  FPIM 
from  service  and  implements  a  statistical  quality  control  plan  to  ensure  appropriate  pro¬ 
cess  performance  and  control.  The  second  alternative  investigates  the  use  of  an  automated 
blade  inspection  system  in  conjunction  with  human  inspectors.  This  semi-automated  sys¬ 
tem  screens  all  blades,  and  removes  the  majority  of  the  good  blades  thus  reducing  the 
inspectors’  workload.  The  final  design  considers  a  fully  automated  inspection  system  that 
required  no  follow-on  human  inspection. 

To  evaluate  the  alternative  system  designs,  a  variety  of  models  were  developed  to 
estimate  system  performance.  For  each  design,  the  models  estimated  the  system’s  crack 
detection  performance  and  the  zissociated  cost  for  that  performance,  as  well  as  the  sys¬ 
tem’s  maintainability,  equivalent  uniform  annual  cost,  surge  capacity,  and  upgradability. 
A  model  that  provides  a  means  to  quantify  a  portion  of  the  cost  of  incorrectly  classifying  a 
blade  was  developed  and  provides  insights  into  the  ramifications  of  incorrectly  classifying 
blades.  A  robotic  system  was  simulated  to  estimate  the  throughput  potential  of  an  auto¬ 
mated  inspection  module.  Experiments  with  a  CCD  camera  validate  the  design  concepts 
for  the  sensor  used  with  the  automated  inspection  systems  and  offer  insights  into  potential 
ways  to  extract  additional  features  from  the  blade  image  to  more  accurately  classify  jet 
engine  blades.  In  addition,  software  was  developed  to  demonstrate  how  the  image  pro¬ 
cessing  and  classification  ability  of  an  automated  system  could  be  improved  compared  to 
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the  present  system.  The  software  also  validates  the  concept  of  using  neural  networks  to 
increase  the  performance  of  an  automated  inspection  system. 

The  analysis  shows  that  the  present  system  using  the  FPIM  does  not  offer  the  best 
overall  performance  because  of  its  high  cost  and  low  maintainability.  The  FPIM  should 
be  removed  from  service  while  an  improved  automated  inspection  system  is  developed. 
The  manual  inspection  process  offers  the  best  performance  for  the  near  future  because 
the  system  is  well  developed  and  there  is  high  confidence  it  can  satisfy  the  inspection 
requirements.  To  be  more  effective,  the  IBIS  process  requires  additional  control  to  quan¬ 
tify  and  monitor  its  performance.  A  semi-automated  inspection  system  is  feasible  with 
current  technology,  but  research  must  be  continued  to  complete  the  system  design  and  de¬ 
velop  efficient  image  processing  and  classification  software.  A  fully  automated  inspection 
system  that  requires  no  human  inspectors  is  also  fe«isible,  but  requires  further  technical 
development.  The  study  demonstrates  that  the  fully  automated  system  could  dramatically 
outperform  the  manual  inspection  process  by  improving  the  consistency  of  the  inspection 
process  and  raising  the  quality  of  the  blades  returned  to  service.  All  efforts  should  be 
made  to  pursue  a  fully  automated  inspection  system  because  it  offers  the  best  potential  to 
reduce  the  costs  associated  with  misclassifying  blades. 

1.3  Thesis  Outline 

Chapter  II  describes  the  current  inspection  system,  presents  background  information 
on  jet  engines,  and  emphasizes  the  problems  with  the  IBIS  and  potential  areas  for  improve¬ 
ment.  The  third  chapter  presents  an  overview  of  the  systems  engineering  approach,  the 
study  objectives,  and  the  evaluation  criteria  as  it  applies  to  the  problem  identified  in  the 
previous  chapter.  Chapter  IV  discusses  the  three  alternative  designs  and  their  expected 
performance  characteristics.  In  chapter  V,  the  alternatives  are  evaluated  to  determine  the 
best  solution  for  the  inspection  process.  The  final  chapter  summarizes  the  study  results 
and  makes  recommendations  on  where  research  should  continue  to  further  improve  the 
inspection  process. 
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1-4  Study  Contributions 


This  study  made  several  noteworthy  contributions  by: 


•  identifying  the  need  for  and  prescribing  a  statistical  quality  control  plan  for  the 
inspection  process 

•  developing  classification  software  to  validate  the  concept  of  using  neural  networks  to 
classify  blades  more  accurately 

•  demonstrating  the  potential  advantages  of  using  charge-coupled  device  (CCD)  cam¬ 
eras  for  data  gathering 

•  providing  a  means  to  quantify  a  portion  of  the  cost  of  incorrectly  classifying  jet  engine 
blades 

•  identifying  a  method  of  using  multiple  images  to  more  accurately  classify  blades 


The  information  resulting  from  these  contributions  provided  useful  insights  in  devel¬ 
oping  as  well  as  evaluating  inspection  systems  utilizing  fluorescent  penetrants. 


II.  Background  Information 


2. 1  Overview 

The  objective  of  this  chapter  is  to  present  the  background  information  pertinent  to 
the  study  of  the  blade  inspection  process  at  San  Antonio  Air  Logistics  Center  (SA-ALC). 
Section  2.1.1  describes  the  role  of  the  SA-ALC  in  the  USAF  jet  engine  inspection  program 
and  provides  some  general  information  about  jet  engines  and  engine  blades.  Following  this 
presentation,  section  2.2  outlines  the  current  inspection  system  and  presents  an  assessment 
of  its  performance  and  potential  problem  areas.  Finally,  section  2.3  presents  assumptions 
about  the  current  inspection  system’s  performance  and  anticipated  increases  in  inspection 
capability,  which  are  used  for  the  remainder  of  the  study. 

2.1.1  SA-ALC’s  Role  in  USAF  Engine  Inspection.  The  SA-ALC  is  responsible  for 
the  depot  level  maintenance  of  all  Pratt  and  Whitney  FlOO  engines  from  aircraft  based  in 
the  continental  United  States  (CONUS).  The  F-15  and  F-16  fighter  aircraft  use  the  FlOO 
style  engines  manufactured  by  Pratt  and  Whitney  (P&W)  and  General  Electric  (GE). 
The  P&W  engines  are  designated  with  an  FlOO  series  number,  while  the  GE  engines  are 
designated  with  an  FllO  number.  Both  types  of  engines  have  similar  performance  char¬ 
acteristics  with  only  minor  differences.  SA-ALC  only  maintains  the  P&W  FlOO  engines. 
The  GE  FllO  engines  are  maintained  at  Oklahoma  City  ALC  and  all  engines  from  aircraft 
stationed  overseas  are  maintained  outside  the  CONUS.  Figure  2.1  displays  the  relationship 
between  SA-ALC  and  the  USAF  engine  inventory.  Each  progressively  smaller  block  in  the 
diagram  represents  a  decrease  in  the  number  of  engines  to  be  considered. 

The  maintenance  process  at  SA-ALC  includes  inspecting  the  engine  components  for 
defects  after  the  engine  has  been  disassembled  to  insure  that  the  parts  are  still  serviceable. 
Each  jet  engine  contains  over  one  thousand  fan,  compressor,  and  turbine  blades  which 
must  be  inspected  for  cracks  and  other  flaws  before  they  are  replaced  in  an  engine.  The 
Integrated  Blade  Inspection  System  (IBIS)  within  SA-ALC  performs  fluorescent  penetrant 
(FP)  inspection  on  engine  blades  to  search  for  cracks.  The  IBIS  also  inspects  turbine 
blades  to  check  for  blocked  cooling  holes.  The  IBIS  contains  the  Fluorescent  Penetrant 
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Figure  2.1.  Relationship  between  USAF  jet  engine  inventory  and  the  SA-ALC  inspection 
process. 
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Inspection  Module  (FPIM)  which  is  a  robotic  system  capable  of  automatically  inspecting 
two  types  of  compressor  blades.  Section  2.2  presents  a  complete  description  of  the  IBIS, 
with  detailed  information  about  the  FPIM,  since  they  are  the  primary  focus  of  this  study. 
The  IBIS  is  mentioned  at  this  point  only  to  show  where  it  fits  into  the  overall  maintenance 
process  within  the  Air  Force. 

The  scope  of  the  systems  engineering  effort  must  be  defined  appropriately  in  order 
for  the  study  to  proceed  efficiently.  Figure  2.1  provides  one  way  of  viewing  how  the 
scope  of  the  problem  can  be  varied  by  considering  different  blocks  within  the  diagram. 
The  smaller  blocks  reflect  a  tighter  problem  scope  because  they  contain  fewer  subsystems 
and,  accordingly,  process  fewer  jet  engine  blades.  The  majority  of  effort  for  this  study 
concentrated  on  ways  to  improve  the  performance  of  the  IBIS.  With  this  in  mind,  the 
relationship  between  the  IBIS  and  the  other  levels  within  figure  2.1  must  be  understood  to 
insure  that  all  factors  influencing  the  problem  are  properly  considered.  The  remainder  of 
this  section  presents  a  variety  of  facts  about  P&W  FlOO  engines,  their  maintenance  history, 
and  how  the  engine  maintenance  influences  the  workload  at  SA-ALC.  Specific  attention  is 
focused  on  the  number  of  engine  blades  requiring  inspection. 

The  first  step  in  understanding  the  maintenance  requirements  of  the  P&W  engines 
is  to  realize  how  many  engines  there  are  in  the  Air  Force  inventory.  Table  2.1  lists  the 
number  of  FlOO  engines  in  the  Air  Force  fleet  and  the  number  of  spare  engines.  Four  types 
of  P&W  FlOO  engines  are  listed  in  the  table,  with  the  type  number  following  the  “F100-” 
in  the  first  column. 


Engine  Type 

Number  in  Fleet 

Number  of  Spares 

FIOO-IOOA/B/C 

1182 

318 

FlOO-200 

796 

195 

FlOO-220  (F-15) 

448 

65 

FlOO-220  (F-16) 

340 

39 

FlOO- 229 

80 

12 

Total 

2846 

629 

Table  2.1.  Number  of  Pratt  and  Whitney  FlOO  engines  in  Air  Force  fleet. 
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SA-ALC’s  workload  is  a  function  of  the  number  of  maintenance  events  required  for 
each  engine  type.  Maintenance  events  are  recorded  according  to  the  type  of  high  pressure 
compressor  (HPC)  module  in  the  engine.  There  are  six  separate  HPC  module  types  for 
P&W  engines.  Table  2.2  outlines  the  relationship  between  the  HPC  modules,  the  engine 
types,  and  the  type  of  aircraft  for  the  engines  that  SA-ALC  services. 


HPC  Module 

Engine  Type 

Aircraft 

FlOO  23B 

FIOO-IOOA 

F-15 

FlOO  24B 

FlOO-200 

F-16 

FlOO  25B 

FlOO-220 

F-15,F-16 

FlOO  26B 

msam 

FlOO  34B 

FIOO-IOOB/C 

F-15 

FlOO  34B 

F-16 

Table  2.2.  Pratt  &  Whitney  FlOO  engine  HPC  modules  in  Air  Force  inventory. 


Using  the  module  numbers  shown  in  table  2.2,  the  actuarial  data  was  consulted  to 
determine  the  number  of  times  the  modules  are  sent  to  the  depot  for  inspection  during  a 
given  year  (62).  A  module  will  be  sent  to  the  depot  as  part  of  a  scheduled  maintenance 
event,  if  its  time-in-service  has  expired,  or  as  an  unscheduled  event,  if  the  module  is 
deemed  as  malfunctioning.  For  example,  an  engine  malfunction  may  be  identified  by  a 
damaged  or  cracked  compressor  case,  low  power  or  thrust,  or  excessive  vibration  or  rough 
operation  (63).  Table  2.3  lists  both  scheduled  and  unscheduled  maintenance  events.  In 
addition,  this  table  presents  an  estimate  of  the  number  of  unscheduled  maintenance  events 
caused  by  compressor  blades,  which  is  relevant  for  this  study.  The  blade  inspection  system 
attempts  to  minimize  the  number  of  flawed  blades  that  return  to  service.  The  number 
of  unscheduled  maintenance  events  caused  by  defective  blades  gives  an  indication  of  the 
quality  of  the  inspection  process. 

Given  the  information  about  the  types  of  engine  modules  and  the  number  of  mainte¬ 
nance  events  per  year,  the  approximate  number  of  engine  blades  requiring  inspection  each 
year  can  be  calculated.  To  perform  the  computation,  the  number  of  blades  per  engine 
must  also  be  known.  Table  2.4  presents  the  number  of  compressor,  turbine,  and  fan  blades 
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Fiscal 

Year 

HPC 

Module 

Scheduled 

Events 

Unscheduled 

Events 

Unsched.  caused 
by  compressor  blades 

1988 

23B 

371 

118 

15 

24B 

278 

104 

12 

1989 

23B 

389 

105 

19 

24B 

315 

111 

18 

1990 

23B 

259 

117 

10 

24B 

204 

158 

21 

25B 

2 

12 

1 

1992 

23B 

58 

47 

6 

34B 

9 

168 

28 

24B 

79 

111 

24 

25B 

4 

53 

4 

Table  2.3.  Scheduled  and  unscheduled  depot  maintenance  events  for  Pratt  &  Whitney 
FlOO  engines.  For  FY88,  FY89,  and  FY90  the  high  pressure  compressor  (HPC) 
module  34B  data  was  unavailable.  FY91  data  was  not  available. 


contained  in  the  P&W  FlOO- 229  engine.  It  is  assumed  that  the  other  FlOO  engine  types 
contain  a  similar  number  of  blades. 


FlOO- 229  Engine  Blades 

Stage 

Number 

Compressor 

742 

Turbine 

292 

Fan 

118 

Total 

1152 

Table  2.4.  Types  and  number  of  engine  blades  in  P&W  FlOO-229  engines. 


Using  the  information  provided  in  tables  2.3  and  2.4,  the  expected  number  of  blades 
per  year  requiring  inspection  at  SA-ALC  can  be  calculated.  The  number  of  scheduled  and 
unscheduled  maintenance  events  are  summed  for  each  year  and  multiplied  by  the  number 
of  blades  per  engine  to  generate  the  data  shown  in  table  2.5. 

The  personnel  at  SA-ALC  estimated  that  the  IBIS  system  processes  approximately  700,000 
compressor  and  turbine  blades  per  year,  which  is  consistent  with  the  information  shown 
in  the  first  two  rows  of  table  2.5  (65).  The  SA-ALC  personnel  also  predicted  that  the 
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Table  2.5.  Expected  number  of  blades  per  year  requiring  inspection  at  SA-ALC.  The 
values  have  been  rounded  to  the  nearest  thousand. 


number  of  blades  requiring  inspection  would  increase  to  approximately  1.2  million  in  the 
neax  future  (65). 

There  are  two  reasons  for  the  increase  in  the  inspection  workload.  First,  SA-ALC 
wants  to  increase  the  number  of  engine  types  it  inspects,  which  would  directly  boost  the 
values  in  table  2.5.  The  second  reason  is  that  the  Air  Force  is  restructuring  its  overall 
maintenance  procedures  which  will  place  an  additional  burden  on  the  depot.  Blades  to  be 
inspected  presently  arrive  at  the  IBIS  facility  as  show  in  figure  2.2.  Conceptionally,  level  1 


L«v*l1 

Itainlwuiiio* 


L«v«l2 

■taiirtviwno* 


D«pol 

(SA-ALC) 


Figure  2.2.  Flow  of  blades  through  two  level  maintenance  system. 

maintenance  is  the  working  environment  on  the  flightline  where  the  engines  are  removed 
and  level  2  is  the  environment  where  field  maintenance  is  performed.  The  Air  Force  is 
presently  changing  to  a  system  as  shown  in  figure  2.3.  Under  this  new  system,  level  1 


D«pot 

(SA-ALC) 


Figure  2.3.  Flow  of  blades  through  one  level  maintenance  system. 

maintenance  shops  will  be  responsible  for  engine  removal  and  minor  maintenance  actions. 
The  depot’s  workload  will  increase  to  compensate  for  the  loss  of  the  level  2  shops. 
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SA-ALC  wants  to  be  able  to  handle  the  expected  increase  in  the  number  of  blades 
requiring  inspection  in  a  cost  effective  manner.  The  inspection  personnel  believe  the  cur¬ 
rent  inspection  system,  including  the  IBIS  and  the  FPIM,  are  performing  adequately,  but 
they  also  have  identified  a  desired  to  improve  the  performance  of  the  system.  An  improved 
system  would  be  better  equipped  to  handle  the  anticipated  increased  workload.  The  follow¬ 
ing  section  describes  how  the  current  system  operates  along  with  some  of  its  performance 
characteristics.  The  description  indicates  the  areas  in  which  the  system  performance  is 
acceptable  as  well  as  areas  where  it  could  be  improved.  The  following  discussion  narrows 
the  scope  of  the  research  effort  to  a  limited  number  of  engine  parts  and  a  single  inspection 
system  for  engine  blades.  As  the  scope  is  limited,  the  above  information  about  engines, 
high  pressure  compressor  modules,  and  the  number  of  blades  must  be  kept  in  mind  to  un¬ 
derstand  how  the  inspection  process  at  SA-ALC  fits  into  the  overall  maintenance  process 
for  the  Air  Force  engine  inventory. 

2.2  Assessment  of  Current  Inspection  System 

2.2.1  Introduction.  The  systems  engineering  design  team  researched  the  present 
inspection  system  by  reviewing  published  literature,  meeting  with  the  original  designers 
of  the  system,  and  performing  a  fact  finding  mission  to  SA-ALC.  The  following  sections 
provide  a  brief  overview  of  the  operation  of  the  current  inspection  system  along  with  a 
discussion  of  problematic  areas. 

2.2.2  Historical  Information.  The  General  Electric  (GE)  Aircraft  Engine  Business 
Group  in  Cincinnati  OH  developed  the  Integrated  Blade  Inspection  System  (IBIS)  through 
Air  Force  contract  F33615-78-C-5095.  The  Materials  Directorate  at  the  Wright  Laboratory 
(called  the  Air  Force  Wright  Aeronautical  Laboratories  (AFWAL)  at  the  time)  initiated  the 
contract  in  September  1978  to  develop  the  IBIS.  GE  was  the  prime  contractor  responsible 
for  designing,  constructing,  delivering,  and  testing  the  inspection  system.  The  original 
IBIS  program  consisted  of  eight  phases  which  developed  the  following  five  subsystems:  the 
Fluorescent  Penetrant  Inspection  Module  (FPIM),  the  Visual  Inspection  Module  (VTM), 
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the  Information  Computer  System  (ICS),  a  Computer  Aided  Design  (CAD)  System,  and 
the  Inspection  Plan  Generation  System  (IPGS). 

GE  developed  the  FPIM  to  inspect  blades  prepared  by  the  Automated  Fluorescent 
Penetrant  Preprocessing  Module  (AFPPM).  The  AFPPM  prepares  a  variety  of  engine 
parts  for  inspection,  including  compressor  blades  and  turbine  blades.  After  the  module 
applies  Fluorescent  Penetrant  (FP),  inspectors  either  manually  inspect  the  parts,  searching 
for  defects,  or  use  the  FPIM  to  automatically  inspect  selected  compressor  blades. 

GE  designed  the  VIM  to  detect  and  measure  visual  flaws  in  airfoils,  but  the  sensor 
depth  of  field  and  the  processing  speed  were  inadequate.  The  Air  Force  suspended  the 
development  of  the  VIM  and  returned  the  unit  to  GE  for  storage. 

The  computer  systems  were  designed  to  collect  statistics  on  the  processed  blades  and 
to  develop  inspection  plans  which  determine  how  to  present  the  blades  to  the  inspection 
sensors.  The  ICS  is  no  longer  part  of  the  system  and  SA-ALC  has  extensively  modified 
the  other  compute  's  since  they  were  first  developed. 

The  Air  Force  augmented  the  original  system  with  the  Infrared  Inspection  Mod¬ 
ule  (IRIM),  which  is  independent  of  the  other  preparation  and  inspection  modules.  The 
IRIM  forces  hot  air  through  turbine  blades  to  locate  and  characterize  the  cooling  holes  to 
determine  if  any  are  blocked. 

In  its  present  form,  the  IBIS  consists  of  only  three  modules:  the  AFPPM,  the  FPIM, 
and  the  IRIM  as  shown  in  figure  2.4.  For  this  design  project,  the  AFPPM  and  the  FPIM 
are  of  primary  importance,  so  they  will  be  discussed  in  detail.  Prior  to  this  discussion,  a 
brief  review  of  the  fluorescent  penetrant  process  is  presented. 

2.2.3  Review  of  the  Fluorescent  Penetrant  Process.  Fluorescent  Penetrant  Inspec¬ 
tion  (FPI)  is  a  nondestructive  method  to  detect  discontinuities  that  are  open  to  the  surface 
of  a  test  article.  Most  FP  application  techniques  use  a  variation  of  the  following  algorithm. 
FP  is  applied  to  the  test  article  and  then  allowed  to  seep  into  discontinuities  that  are  open 
to  the  surface.  After  a  pre-established  amount  of  time,  the  surface  penetrant  is  removed 
and  a  developer  applied  to  enhance  the  bleedout  of  the  penetrant  from  the  discontinuities. 
Following  bleedout,  the  test  article  is  inspected. 
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Bad  Blaaes  Good  Blades 
and 
Parts 

Figure  2.4.  IBIS  Block  Diagram 

From  this  algorithm  there  are  five  critical  steps;  blade  preparation,  application  of 
penetrant,  removal  of  surface  penetrant,  application  of  developer,  and  inspection.  These 
steps  will  be  explained  with  respect  to  this  project.  Consult  references  (37)  or  (55)  for  a 
more  general  treatise  of  FPL 

Preparation  includes  all  the  steps  necessary  to  remove  surface  contaminants  from  the 
test  article.  There  are  two  main  reasons  this  cleaning  must  be  performed.  First,  the  surface 
must  be  clean  enough  to  allow  the  penetrant  to  enter  the  discontinuity.  Second,  penetrant 
will  adhere  to  contaminants  left  on  the  article  appearing  as  background  indications  during 
the  inspection  step.  A  good  cleaning  procedure  uses  detergent  that  does  not  leave  a  residue 
or  water  spots,  which  also  may  increase  background. 
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Application  of  penetrant  can  be  accomplished  in  various  ways,  such  as  electrostatic 
spraying,  dipping  the  test  article  in  a  reservoir  of  penetrant,  or  brushing  on  the  penetrant. 
Regardless  of  the  application  procedure,  the  primary  objective  is  to  coat  the  test  article 
uniformly.  After  application  the  penetrant  is  allowed  to  seep  into  any  flaws.  The  time 
needed  for  the  penetrant  to  seep  in  is  a  function  of  the  type  of  penetrant  used  and  the  size 
of  defects  to  be  detected.  Typical  times  range  between  15  and  75  minutes. 

Removal  of  penetrant  depends  on  the  specific  type  of  penetrant  used,  such  as  water 
washable,  solvent  removable,  or  emulsifiable.  Emulsifiable  penetrants  are  used  to  detect 
small  flaws  and  are  used  by  SA-ALC.  The  amount  of  time  the  emulsifier  is  applied  to  the 
test  article  is  very  important.  Increased  emulsifier  time  decreases  background  but  increases 
the  probability  of  removing  penetrant  from  the  flaw.  Conversely,  short  emulsifier  times  do 
not  remove  the  surface  penetrant  appropriately.  After  the  emulsifier  step,  the  article  is 
rinsed  off  to  prevent  further  penetrant  removal. 

The  developer,  emulsifier,  and  penetrant  chemicals  are  matched  for  optimum  per¬ 
formance,  either  by  the  manufacturer  or  a  penetrant  testing  organization.  Developer  is 
applied  to  aid  the  penetrant  in  wicking  out  of  a  flaw  and  to  increase  the  brightness  of  an  in¬ 
dication.  As  with  penetrants,  there  are  several  different  types  of  developer.  These  include 
dry  powder,  water  soluble,  water  suspendable,  and  nonaqueous  solvent  suspendable. 

Inspection  of  the  test  article  can  be  either  manual  or  automatic.  In  a  manual  oper¬ 
ation,  the  inspector  will  shine  an  ultraviolet  lamp  on  the  test  object  to  excite  the  visible 
dye  in  the  penetrant  resulting  in  the  emission  of  visible  light  in  the  region  of  500  to  600 
nanometers  (yellow  green).  This  band  was  chosen  to  correspond  to  the  maximum  spectral 
response  of  the  human  eye.  The  light  sources  for  automatic  inspection  techniques  vary  de¬ 
pending  on  the  design  of  the  system.  Both  scanning  laser  (9)  and  high  intensity  ultraviolet 
lamps  have  been  used  (1). 

2.S.4  Functional  Overview  of  the  IBIS.  The  AFPPM  is  a  computer  controlled  set 
of  stations  used  to  prepare  aircraft  parts  for  FP  inspection.  A  block  diagram  of  the 
AFPPM  and  preprocessing  stations  is  shown  in  figure  2.5.  Engine  blades  to  be  inspected 
are  placed  in  the  vibratory  where  aluminum  oxide  blocks  abrasively  remove  contaminants. 
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Figure  2.5.  AFPPM  Block  Diagram 


After  vibratory  cleaning,  an  automated  washing  machine  cleans  the  parts  using  La-Ma- 
Di-Wa,  an  industrial  liquid  washing  soap.  After  cleaning,  an  operator  places  the  parts  in 
baskets  to  be  processed  by  the  AFPPM.  Baskets  are  carried  from  station  to  station  by 
an  overhead  conveyor  system  controlled  by  the  AFPPM  computer.  The  computer  also 
controls  the  application  times  of  each  station  via  a  recipe  the  operator  selects.  Application 
times  for  stations  depend  on  specific  station  actions.  For  example,  for  the  FP  application 
block  shown  in  figure  2.5,  the  controlled  time  is  FP  application  time.  The  AFPPM  uses 
an  emulsifiable  penetrant  applied  with  electrostatic  sprayers.  The  excess  penetrant  is 
removed  with  an  emulsifier  and  a  dry  powder  developer  is  used  to  enhance  the  wicking  of 
the  penetrant. 

After  the  basket  of  parts  exit  the  last  station,  approximately  84  minutes  after  the 
basket  enters  the  system,  an  operator  removes  the  basket  from  the  overhead  conveyor 
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system  and  sets  it  aside.  The  parts  then  wait  the  appropriate  development  time,  typically 
30  minutes  to  2  hours,  prior  to  inspection. 

Following  development  time,  an  operator  must  carry  a  basket  of  blades  from  the 
AFPPM  to  either  the  FPIM  or  a  human  inspector.  If  the  FPIM  is  used,  the  operator 
removes  a  blade  from  the  basket  and  passes  it  to  the  FPIM’s  manipulator.  The  FPIM 
manipulates  the  blade  along  with  the  sensor  head  in  accordance  with  a  stored  scan  plan 
to  acquire  indication  data.  After  the  data  is  acquired,  the  FPIM  drops  the  blade  onto  its 
conveyor  belt  which  transports  the  blade  back  to  the  operator.  Depending  on  the  outcome 
of  the  FPIM’s  indication  detection  algorithm,  a  red,  blue,  or  green  light  is  illuminated. 
These  lights  represent  the  following  conditions: 


Green  light:  indicates  no  relevant  indications  were  found,  blade  disposition  good 

Blue  light:  indicates  too  much  background  noise  to  classify,  manual  re-inspection  neces¬ 
sary,  blade  disposition  unknown 

Red  light:  indicates  manual  re-inspection  necessary,  relevant  indications  detected,  blade 
expected  bad. 

After  the  FPIM  makes  its  decision,  the  operator  must  manually  remove  the  blade 
from  the  FPIM’s  conveyor.  If  a  red  or  blue  light  condition  exists,  the  operator  re-inspects 
the  blade  and  then  deposits  it  into  the  appropriate  good  or  bad  outgoing  basket.  If  a  green 
light  condition  exists,  the  operator  deposits  the  blade  into  an  outgoing  good  basket.  From 
the  time  the  operator  loads  a  blade  into  the  FPIM  until  the  time  the  FPIM  displays  its 
disposition  is  approximately  15  seconds. 

2.2.5  Identification  of  IBIS  problem  areas.  As  part  of  this  project,  an  assessment 
of  the  current  system  was  performed.  The  following  sections  provide  a  brief  overview  of 
the  findings  of  this  assessment. 

2.2.5. 1  AFPPM  Shortcomings.  There  is  no  direct  connection  between  the 
AFPPM  and  the  manual  inspector  booths  or  the  FPIM.  This  means  that  the  baskets  must 
be  removed  from  the  over-head  conveyor  system  manually  and  given  to  the  appropriate 
inspector  (human  or  FPIM).  Although  this  is  not  an  overly  burdening  task,  performing  it 
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decreases  the  availability  of  an  operator  or  inspector  from  performing  doing  more  produc¬ 
tive  tasks. 

The  blades  should  be  inspected  30  minutes  to  2  hours  after  the  baskets  leave  the 
AFPPM.  This  time  allows  the  indications  to  develop  properly.  Unfortunately,  the  present 
system  provides  no  assurance  that  the  baskets  are  being  inspected  in  this  time  window. 

Recipes  for  parts  are  selected  based  on  past  experience  and  the  human  inspector’s 
subjective  evaluation  of  blade  quality.  The  chemicals  are  changed  periodically  to  compen¬ 
sate  for  decreases  in  performance  of  chemicals  as  a  function  of  time.  But  since  the  line 
has  not  been  characterized  in  terms  of  how  chemical  conditions  or  recipe  settings  influ¬ 
ence  blade  inspection,  potential  tradeoffs  are  not  available.  This  raises  the  question  as  to 
whether  or  not  all  the  chemicals  need  to  be  changed  periodically  or  just  a  significant  few. 

2.2.5.2  FPIM  Shortcomings.  The  FPIM  contains  computer,  image  process¬ 
ing,  manipulator,  sensor,  and  scan  plan  generation  subsystems.  Each  of  these  subsystems 
were  examined  in  detail.  The  following  sections  provide  a  brief  description  of  each  subsys¬ 
tem’s  function  along  with  identified  problematic  areas. 

2.2. 5. 3  Computer  subsystem.  A  complex  and  highly  interdependent  computer 
system  controls  the  FPIM.  The  computer  subsystem  controls  the  manipulation  of  the  blade 
under  inspection,  accepts  information  from  the  sensor  subsystem,  processes  the  informa¬ 
tion,  and  passes  judgement  on  the  inspected  part.  Three  Digital  Equipment  Corporation 
(DEC)  PDP-11/23  computers  manage  the  entire  process  and  perform  high-level  part  as¬ 
sessment.  These  three  computers  are  the  Mzister  Computer  and  two  slave  computers  called 
the  Sensor  and  Manipulator  Computers.  The  Master  Computer  serves  as  the  inspection 
process  controller  and  the  flaw  report  generator.  The  Sensor  Computer  monitors  the  FPIM 
preprocessor  performance  and  performs  limited  image  processing  on  data  received  from  the 
preprocessor.  The  Manipulator  Computer  controls  the  thirteen  servo  axes  that  comprise 
the  manipulator  subsystem.  Intermediate  between  the  DEC  PDP-ll/23s  and  the  sensor 
subsystem  exists  the  special  purpose  FPIM  preprocessor.  This  unique  hardwired  system 
receives  the  signals  from  the  photomultiplier  tube,  performs  significant  processing  on  the 
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data,  passes  the  information  to  the  sensor  computer,  monitors  the  inspection  process,  no¬ 
tifies  the  sensor  computer  of  attained  milestones,  and  coordinates  the  operation  of  the 
manipulator  and  sensor  computers. 

The  computer  subsystem  emerges  as  a  significant  weakness  of  the  FPIM.  The  special 
purpose  preprocessor  represents  the  primary  shortcoming  of  the  computer  subsystem.  It 
is  a  nonstandard,  hardwired  processor  that  performs  the  bulk  of  the  data  processing.  The 
one-of-a-kind  design  presents  a  serious  maintenance  problem.  Diagnostic  and  repair  tasks 
are  difficult,  and  replacement  components  are  not  readily  available. 

In  addition  to  maintainability  problems,  the  computer  subsystem  makes  system  up¬ 
grades  virtually  impossible.  The  unique  nature  of  the  FPIM  preprocessor  required  complex 
module  interconnections  when  it  was  developed.  Both  the  FPIM  software  and  hardware 
expect  the  system  to  function  in  its  unique  manner  and  cannot  handle  even  simple  modifica¬ 
tions.  Any  changes  to  the  system  would  have  to  exactly  emulate  the  current  configuration. 
For  this  reason,  performance  improvements  to  the  current  FPIM  system  are  infeasible. 

2. 2. 5. 4  Image  Processing  subsystem.  The  image  processing  subsystem  is  re¬ 
sponsible  for  making  the  decision  about  blade  quality  by  interpreting  indications.  To  do 
this,  the  FPIM  relies  on  a  software  algorithm  written  in  Fortran. 

The  FPIM  software  is  highly  machine  dependent  because  of  the  way  data  is  stored 
and  manipulated.  The  image  processing  and  classification  software  receives  raw  pixel 
data  from  the  sensor  and  determines  if  the  indications  on  the  blade  surface  are  significant 
enough  to  reject  the  blade.  The  system  is  very  accurate  when  labeling  blades  that  do  not 
have  defects.  In  other  words,  if  the  blade  is  labeled  good,  there  is  approximately  a  99% 
probability  that  the  blade  is  actually  good  (65).  If  a  blade  has  a  large  indication  or  many 
small  indications,  the  FPIM  memory  buffers  can  fill  and  the  system  may  not  be  able  to 
make  a  determination  within  the  allowed  15  second  time  window.  Since  the  memory  and 
processing  speed  limitations  prevent  the  system  from  accurately  cl2issifying  the  blade,  it 
labels  the  blade  suspect  by  default,  so  a  human  inspector  will  check  it. 

Even  when  the  pixel  data  is  within  the  memory  limits  of  the  FPIM  processor,  the 
system  still  has  trouble  accurately  identifying  cracks.  The  main  problem  is  that  the  decision 
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function  uses  a  rule  based  classification  scheme  that  only  considers  the  number  of  pixels 
and  the  maximum  pixel  intensity  in  each  indication.  The  original  designers  selected  these 
two  features  based  on  empirical  data  that  showed  most  cracks  could  be  identified  with  this 
limited  amount  of  information  (51). 

Cracks  and  other  types  of  flaw  indications  create  complex  FP  patterns  on  the  blade 
surface.  A  human  inspector  uses  a  variety  of  information  about  the  crack  indications  to 
accurately  identify  them.  Indication  size  and  intensity  are  important,  but  the  shape  of  the 
indication,  how  linear  the  indication  is,  and  the  way  the  intensity  of  the  indication  changes 
across  the  surface  are  all  important  factors  that  help  distinguish  cracks  from  non-crack 
indications  (65).  A  system  that  extracts  more  information  from  each  indication  than  the 
FPIM  and  then  processes  the  information  efficiently  to  make  a  decision,  will  likely  be  able 
to  more  accurately  distinguish  between  good  and  defective  blades. 

Since  it  requires  information  about  two  indication  features,  the  FPIM  only  performs 
a  small  amount  of  image  processing.  The  present  image  processing  operations  include 
dynamic  background  removal  and  thresholding  to  remove  noise  and  dim  pixels  from  the 
data  set.  The  system  applies  the  threshold  uniformly  to  all  data,  so  it  has  no  way  to 
compensate  for  blades  that  may  have  slightly  different  average  intensity  levels.  The  system 
binds  pixels  together  and  grows  regions  which  represent  separate  indications.  At  this  stage 
of  processing,  each  pixel  is  8  mils  by  8  mils  in  size.  The  FPIM  also  eliminates  indications 
with  fewer  than  four  pixels  from  the  data  set  because  they  are  considered  too  small  to  be 
significant. 

The  FPIM  software  performs  its  function  marginally  well  and  there  is  clearly  room 
for  improvement.  The  feature  set  is  limited  because  it  does  not  extract  enough  informa¬ 
tion  from  each  indication.  In  addition,  the  image  processing  routines  are  not  sufficiently 
flexible  to  adapt  to  the  variety  of  blade  conditions  that  may  be  presented  to  the  system. 
The  baseline  software  must  be  enhanced  to  collect  additional  features  about  the  indica¬ 
tions  and  process  them  more  effectively,  so  a  more  accurate  classification  function  can  be 
implemented. 
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2.2.5. 5  Manipulator  subsystem.  As  previously  mentioned,  the  manipulator 
subsystem  is  responsible  for  manipulating  the  blade  and  sensor  head  in  accordance  with 
a  scan  plan.  The  FPIM  manipulators  are  rigidly  configured,  do  not  follow  industry  stan¬ 
dards,  and  require  specialized  maintenance.  These  deficiencies  cause  two  problems.  First, 
upgrading  the  system  to  inspect  additional  blade  types  is  extremely  difficult  because  the 
manipulators  have  limited  work  volume.  Second,  replacement  parts  are  not  readily  avail¬ 
able  and  hence  maintaining  the  system  is  difficult  because  the  control  electronics  were 
custom  made.  In  addition,  the  inflexibility  of  the  manipulator  subsystem  requires  a  man¬ 
ual  operator  to  load  and  unload  the  FPIM  preventing  that  person  away  accomplishing 
more  productive  tasks. 

2.2.5. 6  Sensor  subsystem.  The  sensor  subsystem  is  responsible  for  acquiring 
FP  indication  data  and  transferring  it  to  the  image  processing  subsystem.  The  current 
sensor  subsystem  consists  of  a  laser,  mirrors,  photomultiplier  tube  (PMT),  lens  array,  and 
filters.  The  lens  and  filter  design  results  in  a  PMT  look  angle  of  15  degrees.  This  look 
angle  produces  an  inspection  area  of  .004  by  .004  inches.  Blade  inspection  is  accomplished 
by  using  a  scanning  mirror  to  sweep  the  laser  beam  across  the  blade.  A  receiving  lens 
is  synchronously  scanned  with  the  laser  beam  to  collect  light  from  any  indications.  In 
addition  to  mirror  movement,  the  manipulator  subsystem  moves  the  blade  at  a  fixed  rate 
to  keep  the  area  of  inspection  approximately  normal  to  the  collecting  lens.  Normality  is 
maintained  to  compensate  for  the  apparent  change  in  pixel  size  when  inspecting  the  blade 
at  an  angle.  Light  exiting  the  collection  lens  is  piped  into  the  PMT.  The  output  of  the 
PMT  is  digitized  and  data  bound  together  resulting  in  a  final  pixel  size  of  .008  by  .008 
inches. 

The  procedure  to  synchronously  move  the  blade,  scanning  mirror,  and  collection 
lenses  is  called  a  scan  plan.  Presently,  the  scan  plan  is  very  complex  due  to  the  need  to 
scan  the  entire  surface  of  the  blade  pixel  by  pixel. 

The  current  laser  operates  at  a  wavelength  of  441.6  nanometers.  To  compensate  for 
the  mismatch  between  this  wavelength  and  the  optimal  excitation  wavelength  of  the  FP, 
which  is  365  nanometers,  the  output  of  the  laser  was  increased.  The  power  level  is  several 


2-16 


orders  of  magnitude  greater  than  necessary  had  the  correct  wavelength  been  used.  This 
increased  laser  power  needlessly  complicates  the  filtering  system  and  decreases  the  signal 
to  noise  ratio. 

2.2. 5.7  Scan  Plan  Generation  subsystem.  The  scan  plan  generation  subsys¬ 
tem  provides  the  commands  for  the  manipulator  and  sensor  subsystems  during  blade  in¬ 
spection.  Generation  of  working  scan  plans  requires  several  months,  with  the  majority 
of  the  time  spent  building  three  dimensional  electronic  models  of  a  blade.  The  computer 
hardware  and  software  available  for  this  task  is  outdated  which  contributes  to  the  long 
development  time.  Because  of  the  difficulty  in  generating  scan  plans,  presently  they  exist 
only  for  stage  4  and  9  compressor  blades. 

2.2.6  Additional  Concerns.  In  its  present  configuration,  the  FPIM’s  false  call  rate 
is  50%,  which  means  it  labels  half  of  the  good  blades  as  suspect.  The  false  calls  require 
additional  manual  inspection  which  increases  the  workload  of  the  inspectors.  Contributing 
to  this  high  false  call  rate  is  the  throughput  concern  of  the  original  design.  Blades  are 
assumed  to  be  inspectable  in  15  seconds.  If  a  decision  cannot  be  reached  in  this  time 
because  of  too  many  indications,  the  FPIM’s  decision  algorithm  terminates  yielding  a  blue 
light  condition. 

During  a  visit  to  the  inspection  facility,  the  Design  Team  had  the  opportunity  to 
talk  to  IBIS  personnel  responsible  for  operation  and  support  of  the  FPIM.  IBIS  personnel 
voiced  the  three  following  concerns:  the  system  is  difficult  to  use,  the  FPIM  is  expensive 
to  maintain,  and  the  system  is  difficult  to  repair. 

2.3  Assumptions 

Many  of  the  performance  specifications  and  jet  engine  blade  statistics  used  in  this 
section  have  not  been  confirmed.  In  several  cases  data  is  not  available  or  is  impossible  to 
substantiate  without  extensive  research  effort  which  is  beyond  the  scope  of  this  project.  In 
order  to  proceed  with  the  design  study,  the  following  assumptions  concerning  the  current 
inspection  process  were  made. 


2-17 


1.  The  inspection  system  will  eventually  be  required  to  inspect  1.2  million  blades  per 
year.  This  figure  is  substantially  larger  than  the  present  number  of  blades  inspected, 
but  it  is  in  line  with  the  anticipated  number  of  blades  in  future  years  according  to 
SA-ALC. 

2.  Five  percent  of  all  blades  are  actually  defective.  The  management  at  SA-ALC  made 
this  estimate  based  on  the  percentage  of  parts  identified  as  defective. 

3.  A  human  inspector  requires  5  seconds  to  inspect  a  good  blade  and  30  seconds  to 
inspect  a  bad  blade.  These  times  are  estimates  based  on  observing  an  inspector  in 
action  and  feedback  from  SA-ALC. 

4.  The  FPIM  receives  approximately  5%  of  the  parts  processed  through  the  AFPPM. 
SA-ALC  provided  data  for  calendar  year  1991  showing  that  3.41%  of  the  parts  pro¬ 
cessed  through  the  AFPPM  were  sent  to  the  FPIM.  The  value  5%  was  selected  for 
the  calculations,  because,  according  to  SA-ALC,  the  percentage  of  blades  sent  to  the 
FPIM  in  1991  was  lower  than  average. 

5.  The  FPIM  has  a  50%  probability  of  labeling  a  good  blade  as  defective  (false  call)  and 
a  1%  probability  of  labeling  a  cracked  blade  as  good  (missed  detection).  The  1991 
data  showed  that  the  FPIM  labeled  58.48%  of  all  blades  as  suspect.  Personnel  at 
SA-ALC  estimated  that  the  FPIM  detects  99%  of  aU  bad  blades.  Therefore,  50% 
and  1%  represent  reasonable  estimates  for  the  false  call  and  missed  detection  error 
rates. 

6.  Finally,  all  calculations  assume  all  blades  are  properly  processed  for  inspection.  In 
other  words,  a  human  inspector  can  successfully  process  each  blade  without  repro¬ 
cessing  it  through  the  AFPPM. 


These  assumptions  impact  analysis  throughout  the  study  and  form  the  basis  to  com¬ 
pare  the  baseline  system  to  other  systems.  Additionally,  the  assumptions  provide  the  basic 
foundation  to  develop  future  inspection  systems. 


2.4  Summary 

Even  though  the  IBIS  offered  a  state-of-the-art  solution  to  blade  inspection  at  the 
time  of  its  development,  it  is  not  performing  satisfactorily  at  this  time.  Prior  to  using  the 
IBIS,  inspectors  were  not  finding  as  many  relevant  indications  on  stage  4  and  9  compressor 
blades  (51,  65).  SA-ALC  desires  added  detection  capability  to  decrease  the  number  of 
bad  blades  leaving  the  depot.  This  desire  drives  the  need  for  automatic  blade  inspection. 
However,  as  attested  to  by  SA-ALC  managers,  additional  blade  inspection  capability  is  only 
valuable  if  the  shortcomings  of  the  system  are  manageable.  The  majority  of  the  FPIM’s 
shortcomings  are  due  to  it  being  a  prototype  design  cast  into  an  operational  role.  Several 


2-18 


concerns,  such  as  maintainability  and  upgradability  were  not  addressed  in  the  original 
design.  This  left  SA-ALC  with  a  system  that  does  not  meet  their  inspection  needs.  Any 
second  generation  automatic  blade  reader  should  address  the  identified  shortcomings  of 
the  FPIM.  Table  2.6  summarizes  the  components  and  the  performance  specifications  of 
the  current  system. 


Component 

Quantity 

AFPPM 

1 

ABR 

-  Manipulator  Sys 

1 

-  Sensor 

1 

-  Master  Computer 

1 

Human  Inspectors 

3 

SQC 

-  Inspectors 

0 

-  Microscope  and  Blade  counter 

0 

Performance 

-  False  call  rate 

2.5  percent 

-  Avg  Inspection  Time 

6.88  sec/blade 

-  Probability  ID  Crack 

90  percent 

Table  2.6.  Summary  of  baseline. 


III.  Systems  Engineering  Approach 


S.  1  Introduction 

The  fundamental  goal  for  this  study  was  to  improve  the  jet  engine  blade  inspection 
process  at  SA-ALC.  The  current  inspection  system  has  been  examined  and  its  performance 
characterized  to  identify  problem  areas.  In  addition,  the  personnel  at  SA-ALC  have  de¬ 
scribed  what  they  expect  from  future  inspection  systems.  Using  this  information  as  a 
guide,  the  research  team  developed  alternative  system  designs  that  could  potentially  im¬ 
prove  the  inspection  process.  To  be  efficient,  the  search  for  candidate  solutions  required 
boundaries  to  limit  the  problem,  as  well  as  criteria  to  evaluate  each  solution  to  determine 
its  potential  performance. 

The  systems  engineering  approach  provides  a  methodology  to  examine  complex  prob¬ 
lems,  develop  a  variety  of  potential  solutions,  and  evaluate  the  alternatives  to  determine 
the  best  course  of  action.  One  of  the  keys  behind  a  successful  systems  engineering  design 
project  is  to  completely  define  the  problem  and  to  establish  a  well  integrated  plan  to  use 
as  a  guide  throughout  the  process.  To  better  organize  the  study,  the  team  applied  Unified 
Program  Planning,  which  is  one  of  several  classic  systems  engineering  techniques. 

Hill  and  Warfield  developed  Unified  Program  Planning  as  one  approach  to  initiate  the 
systems  engineering  process  (27).  Their  method  uses  Hall’s  activity  matrix  as  a  framework 
to  logically  partition  a  problem  and  outline  the  steps  necessary  to  approach  a  solution  (24). 
Hall’s  matrix  captures  the  fundamental  aspects  of  systems  engineering  in  a  concise  format 
by  showing  how  to  divide  the  problem  into  logical  steps  for  each  phase  of  the  project.  The 
first  phase  of  the  study  is  program  planning,  which  Hill  and  Warfield  explored  in  detail. 

Hill  and  Warfield  explained  how  to  apply  the  first  three  steps  in  Hall’s  matrix  to 
the  program  planning  phase.  The  three  steps  are  the  problem  definition,  the  value  system 
design,  and  the  system  synthesis.  The  problem  definition  clearly  outlines  what  the  systems 
engineering  study  will  accomplish.  The  value  system  design  establishes  criteria  used  to 
evaluate  alternatives.  Finally,  the  system  synthesis  is  the  process  of  developing  alternative 
designs  that  can  potentially  solve  the  problem. 
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The  Unified  Program  Planning  approach  provides  a  structure  to  relate  all  of  the  ele¬ 
ments  generated  during  the  problem  definition,  value  system  design,  and  systems  synthesis. 
By  monitoring  the  relationships  between  the  various  factors,  the  systems  engineering  study 
can  proceed  in  an  organized  and  efficient  manner.  Sage  offers  a  detailed  description  of  the 
method  outlined  by  Hill  and  Warfield  (53).  The  design  team  followed  Sage’s  approach  to 
fully  define  the  problems  associated  with  the  blade  inspection  process  and  to  efficiently 
search  for  solutions. 

3.2  Program  Planning 

Program  planning  initiates  the  systems  engineering  study  by  describing  the  needs, 
alterables,  and  constraints  associated  with  the  design  problem.  These  factors  specifically 
identify  what  the  study  must  accomplish,  what  elements  contribute  to  the  potential  solu¬ 
tions,  and  the  problem  boundaries. 

5. 2. 1  Needs.  Needs  arise  when  the  present  system  is  not  performing  satisfactorily  or 
the  possibility  of  improvement  has  been  identified.  A  need  can  be  either  a  requirement  that 
must  be  met  or  a  desired  benefit  that  is  not  necessarily  essential  for  system  performance. 
The  needs  identified  in  this  study  are  what  SA-ALC  requires  or  expects  from  future  blade 
inspection  systems  and  are  listed  below. 

1.  Prepare  blades  for  inspection  with  proper  cleaning  and  penetrant  application 

2.  Identify  cracked  blades 

3.  Create  a  maintainable,  reliable,  easy  to  use  inspection  process 

4.  Develop  a  cost  effective  inspection  system 

5.  Allow  for  an  expandable  and  upgradable  system 

6.  Inspect  all  types  of  jet  engine  compressor  and  turbine  blades 

7.  Collect  accurate,  quantitative  measures  of  the  number  and  types  of  blades  processed 
and  their  disposition  after  inspection 

8.  Decrease  human  inspector  workload 


3.2.2  Alterables.  The  alterables  are  the  variables  in  the  inspection  system  that 
can  be  changed  to  develop  solutions  that  satisfy  the  needs.  Alterables  are  factors  that 
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either  can  be  controlled  during  the  study  to  optimize  solutions  or  elements  in  the  externad 
environment  that  are  impossible  to  control,  but  are  subject  to  change.  The  external  factors 
still  influence  the  results  of  the  study,  so  they  must  be  identified  to  account  for  all  of  the 
forces  that  affect  the  design  process.  The  following  alterables  were  considered  to  generate 
candidate  solutions. 

1.  The  inspection  configuration 

2.  Number  of  parts  inspected  manually 

3.  Penetrant  application  procedure  during  blade  preparation 

4.  Image  processing  and  classification  algorithms 

5.  Blade  manipulation  and  presentation  technique 

6.  Computer  hardware  for  data  processing 

7.  Sensor  subsystem 

3.2.3  Constraints.  The  constraints  define  the  limits  on  how  much  the  alterables 
can  be  varied  while  exploring  solutions.  These  factors  also  describe  the  range  of  potential 
values  over  which  the  needs  can  be  satisfied.  The  constraints  identified  below  provide 
reasonable  boundaries  for  solving  the  blade  inspection  problem. 

1.  Use  fluorescent  penetrant  processing  to  prepare  the  blades  for  inspection 

2.  The  proposed  system  will  follow  industry  standards,  where  possible,  or  minimize  risk 
in  developing  other  subsystems 

3.  The  system  must  be  modular  and  upgradable 

4.  The  system  must  satisfy  the  inspection  requirements  outlined  in  the  Air  Force  spec¬ 
ifications,  standards,  and  technical  orders  (15,  14) 

5.  The  system  must  be  capable  of  processing  1.2  million  blades  per  year 

6.  The  system  must  not  introduce  any  new  environmental  hazards 


The  needs,  alterables,  and  constraints  identified  in  the  program  planning  phase  pro¬ 
vided  the  initial  structure  for  the  problem  and  outline  how  potential  solutions  may  be 
generated.  These  three  elements  are  closely  interrelated  and  the  proposed  designs  will 
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require  trade-offs  between  the  various  alterables  in  order  to  satisfy  the  needs  without  vio¬ 
lating  the  constraints.  Once  feasible  solutions  were  developed,  the  value  system  wa.s  used  to 
compare  the  alternatives  and  determine  the  best  solution  to  the  blade  inspection  problem. 

3.3  Value  System 

The  value  system  provides  a  set  of  objectives  for  the  design  problem,  determines  the 
interactions  between  them,  and  organizes  the  objectives  in  a  hierarchical  structure  (53). 
Once  the  objectives  are  organized,  a  set  of  measures  can  be  developed  to  determine  how 
well  each  objective  was  met.  The  hierarchical  structure  provides  information  on  how  to 
weight  the  various  objective  measures  to  calculate  a  performance  value  for  each  alternative. 
Each  design  alternative  is  compared  to  the  objective  measures  to  determine  its  relative 
performance  in  solving  the  problem. 

3.3.1  Objectives.  The  primary  goal  of  the  systems  engineering  design  study  is  to 
improve  the  performance  of  the  blade  inspection  process  at  SA-ALC.  The  following  objec¬ 
tives  were  identified  as  the  essential  elements  in  achieving  the  fundamental  goal. 

1.  To  minimize  the  cost  of  incorrectly  classifying  blades 

2.  To  maximize  maintainability,  reliability,  and  availability 

3.  To  minimize  the  annual  cost  of  the  inspection  process 

4.  To  maximize  the  throughput  capacity 

5.  To  maximize  upgradability 

To  be  valid,  the  objectives  must  be  related  to  the  needs  of  the  design  study.  Together, 
these  five  high  level  objectives  address  all  of  the  needs  identified  in  the  previous  section. 
The  following  brief  discussion  outlines  the  role  of  each  objective  in  the  study. 

Cost  of  incorrectly  classifying  blades.  The  inspection  system  must  have  a  high  prob¬ 
ability  of  identifying  cracked  blades.  If  a  cracked  blade  is  allowed  to  return  to  service, 
there  is  a  chance  it  could  cause  engine  damage  or  additional  maintenance  actions.  On  the 
other  hand,  the  inspection  system  must  also  avoid  labeling  good  blades  as  defective,  which 
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would  prematurely  remove  the  parts  from  service.  Each  type  of  inspection  error  has  an 
associated  cost.  The  inspection  system’s  performance  can  be  quantified  by  determining 
the  expected  cost  of  allowing  a  cracked  blade  to  return  to  service  and  the  cost  of  discarding 
a  good  blade.  Quantifying  the  probabilities  for  each  type  of  error  and  the  associated  costs 
will  provide  a  clear  method  for  measuring  how  well  the  system  satisfies  the  need  to  identify 
cracked  blades. 

Maintainability.  The  system’s  maintainability  directly  influences  its  operating  cost. 
A  maintainable  system  is  easy  to  diagnose  and  repair,  has  efficient  preventive  maintenance 
tasks,  and  does  not  place  a  large  burden  on  the  system  maintenance  team.  The  system’s 
maintainability  also  represents  an  integral  part  of  the  system’s  availability.  The  less  the 
system  is  down  for  maintenance,  the  more  the  system  is  available  for  use.  SA-ALC  requires 
an  available  inspection  system  to  insure  it  can  inspect  jet  engine  blades  to  meet  Air  Force 
requirements. 

Annual  cost.  All  design  alternatives  must  minimize  the  expected  cost  of  the  process 
of  inspecting  blades.  Uniform  annual  cost  was  used  to  capture  the  importance  of  both 
capital  and  annual  operating  costs  of  the  system  designs.  With  a  shrinking  defense  budget, 
SA-ALC  must  make  every  effort  to  maintain  a  cost  effective  process. 

Throughput  capacity.  The  speed  at  which  the  system  can  inspect  blades  determines 
its  capacity  to  expand  in  the  future,  as  well  as  how  the  system  adapts  to  handle  periodic 
surges  of  blades.  The  number  of  different  types  of  blades  requiring  inspection  may  increase 
and  place  greater  demands  on  the  inspection  system  as  the  Air  Force  develops  new  engines. 
In  addition,  a  system  with  high  throughput  requires  less  operating  time  to  process  a  given 
number  of  blades,  which  helps  reduce  the  total  operating  cost. 

Upgradability.  Having  learned  from  mistakes  with  the  FPIM,  the  management  at 
SA-ALC  will  only  accept  an  inspection  system  design  that  can  be  upgraded  in  the  future. 
The  upgradability  objective  simultaneously  addresses  a  constraint  and  offers  a  way  to 
differentiate  between  alternative  designs. 
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3.3.2  Objective  Measures.  The  design  alternatives  will  satisfy  each  of  the  five  high 
level  objectives  to  varying  degrees.  To  help  quantify  how  well  each  design  fulfills  the 
objectives,  the  following  objective  measures  were  identified. 

1.  The  expected  number  of  engine  failures  and  unscheduled  maintenance  events  from 
missed  detection  of  defective  blades 

2.  The  number  of  inspection  errors  throughout  the  process 

3.  The  number  of  blades  per  hour  the  system  is  capable  of  processing 

4.  The  number  of  personnel  and  their  training  required  to  maintain  the  system 

5.  The  functional  division  of  system  hardware  and  software  into  logical  and  self-contained 
modules  or  units 

6.  The  ability  of  the  system  to  perform  self-diagnosis 

7.  The  ability  of  the  system  to  expand  the  number  of  engine  types  that  can  be  inspected 

8.  The  ability  of  the  system  to  increase  the  number  of  types  of  blades  that  can  be 
inspected 

9.  The  annual  operating  costs 

10.  The  system  capital  costs 


Several  design  variables  directly  influence  the  objective  measures.  These  design  vari¬ 
ables  are  the  factors  that,  if  changed,  would  affect  the  values  of  the  objective  measures 
thus  altering  the  results  of  the  analysis.  Knowing  what  these  factors  are  for  each  criterion 
is  important,  but,  for  a  systems  engineering  study,  it  is  also  important  to  understand  how 
the  design  variables  are  interrelated  between  the  criteria.  Table  3.1  expresses  a  simple 
relationship  between  the  design  variables  and  the  five  criteria.  Each  “X”  in  the  table 
means  the  design  variable  listed  on  the  left  influences  the  criterion  listed  at  the  top  of  each 
column. 

Table  3.1  can  be  considered  as  an  interaction  matrix,  using  the  terminology  developed 
by  Hill  and  Warfield  (27).  Interactions  between  the  criteria  occur  when  there  is  more  than 
one  “X”  on  a  particular  row.  The  interactions  indicate  which  criteria  must  be  considered 
together  when  completing  the  analysis. 

Some  of  the  entries  in  the  table  are  not  independent  of  each  other.  For  example,  the 
time  required  for  the  automatic  inspection  will  depend  directly  on  the  manipulator  speed. 
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Factois 


Misclass.  I  Maintain  {  Cost  Throughput  Upgrade 


X 


Number  human  inspectors 


Blades  per  year 


Percent  bad  blades 


Manual  inspection  time  (good  blade) 


Manual  inspection  time  (bad  blade) 


Percent  blades  auto  inspect 


Auto  inspect  time 


Human  false  call  rate 


ABR  false  call  rate 


Number  hours  of  operation 


Human  inspection  hours 


Types  of  blades 


Types  of  engines 


Computer  integration 


Modularity 


System  configuration 


New  system  component  costs 


Software 


Consumable  products 


Spare  parts 


Human  probability  of  detection 


ABR  probability  of  detection 


Blades  inspected  per  hour 


ABR  inspection  hours 


Cost  of  maintenance  event 


Number  of  blades  per  engine 


Cost  of  a  blade 


Percent  good  blades  called  suspect 


Percent  bad  blades  called  suspect 


Number  of  maintenance  personnel 


Training  level  of  maint.  personnel 


Current  maintenance  contract  cost 


Percent  of  maintenance  cost  to 
FPIM,  AFPPM,  and  IRIM 


Cost  of  technicians  per  hour 


Man-years  for  non-inspection  tasks 


Interest  rate 


Life  of  system 


Table  3.1.  Interaction  of  design  variables  and  criteria. 
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XXX 


CCD  exposure  time,  and  the  computation  time.  Some  of  the  information  in  the  table  may 
be  redundant,  but,  since  it  improves  the  understanding  of  how  design  variables  influence 
the  criteria,  it  remains  in  the  table. 

The  information  contained  in  table  3.1  is  an  important  aid  in  the  overall  evaluation 
process  because,  if  there  is  change  in  one  of  the  design  variables,  the  table  immediately 
shows  which  criteria  are  affected.  The  table  made  the  systems  engineering  process  more 
efficient,  since  design  variables  changed  frequently  throughout  the  course  of  the  study. 

The  decision  analysis  in  chapter  V  will  elaborate  on  the  objective  measures  and  apply 
them  to  each  alternative  to  determine  the  best  systems  solution.  The  actual  evaluation 
technique  is  based  on  the  approach  outlined  by  Athey  and  accounts  for  the  relative  impor¬ 
tance  of  the  different  objectives  as  well  as  the  amount  of  confidence  in  the  design  values  (2). 
Figure  3.1  shows  the  relationship  between  the  objectives  and  the  objective  measures.  The 
objectives  are  shown  in  the  rectangles  and  the  measures  are  shown  in  the  the  boxes  with 
rounded  corners. 

Program  planning  and  the  value  system  are  the  initial  steps  taken  to  decompose 
the  design  problem  into  manageable  units.  These  two  developmental  phjises  are  critical 
because  they  determine  what  questions  are  asked  and  the  type  of  answers  that  can  be  gen¬ 
erated  during  the  course  of  the  design  study.  The  needs,  alterables,  and  constraints,  along 
with  the  objectives  and  objective  measures,  provide  a  framework  in  which  solutions  are 
developed  and  evaluated.  The  above  problem  formulation  has  evolved  over  time  through 
several  iterations  to  reach  an  acceptable  structure  that  allows  the  remainder  of  the  systems 
engineering  process  to  continue. 
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Figure  3.1.  Relationship  between  objectives  and  objective  measures. 
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IV.  Alternatives 


After  successful  development  of  the  problem  definition  and  value  system  design  ex¬ 
plained  in  chapter  III,  the  next  step  in  the  unified  program  plan  (UPP)  is  system  synthesis. 
System  synthesis  involves  the  creation  of  alternative  solutions  to  the  problem  that  address 
the  needs  of  the  users  by  varying  the  alterables  of  the  system  without  violating  the  con¬ 
straints  identified  in  the  problem  definition.  In  developing  the  alternatives,  functional 
interaction  blocks  were  used  to  identify  system  component  interactions.  These  blocks 
helped  to  identify  the  significant  design  inputs  and  outputs  of  each  component  area  as 
weU  as  the  functional  flow  of  the  designs.  The  functional  interaction  blocks  used  in  the 
development  of  the  alternatives  can  be  found  in  appendix  A. 

Athey  offers  four  ways  to  develop  system  designs:  use  the  existing  system  design, 
modify  the  existing  system  design,  use  prepackaged  designs,  and  create  new  system  de¬ 
signs  (2).  A  combination  of  the  four  were  used  to  develop  the  alternative  designs.  This 
section  provides  an  explanation  of  the  alternatives  and  representative  configurations  that 
will  be  used  for  analysis.  The  specifics  of  each  alternative  design  are  presented  only  as  a 
means  of  substantiating  the  design  concepts  and  to  establish  a  configuration  with  which  to 
collect  data  for  analysis.  The  underlying  difference  between  the  alternatives  is  the  degree 
of  human  intervention  in  the  inspection  process.  The  three  alternatives  are  summarized 
as  manual  inspection,  semi-automated  inspection,  and  fully  automated  inspection. 

4-1  Alternative  1:  Manual  Inspection 

4-1.1  Overview.  Alternative  1  is  a  modified  form  of  the  existing  system.  This  alter¬ 
native  is  the  optimum  alternative  for  low  capital  cost.  The  difference  between  alternative  1 
and  the  current  system  is  the  removal  of  the  FPIM  from  the  system  and  the  addition  of 
a  Statisical  Qualtiy  Control  (SQC)  plan.  Chapter  II  provided  a  discussion  of  the  current 
system.  In  this  section  the  focus  will  be  on  the  SQC  plan. 

Based  on  1991  data  provided  by  SA-ALC  and  mentioned  previously  in  chapter  II, 
the  FPIM  is  currently  used  to  process  less  than  5%  of  the  jet  engine  blades  inspected  by 
the  IBIS.  However,  if  the  maintenance  contract  for  the  IBIS  is  equally  divided  among  its 
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three  major  components,  then  the  FPIM’s  maintenance  costs  account  for  33%  of  the  total 
maintenance  cost.  The  consistency  and  resolution  advantages  offered  by  the  FPIM  are 
negated  by  its  maintainability  problems  and  its  high  false  call  rate.  Recognizing  that  the 
majority  of  the  problems  with  the  current  system  stem  from  shortcomings  of  the  FPIM 
and  that  the  FPIM  is  not  easily  upgradable,  removing  the  FPIM  from  the  system  is  a 
logical  decision.  The  SQC  plan  is  aimed  at  providing  insights  into  how  the  system  is 
performing  and  how  the  system  can  be  improved.  Currently,  there  is  little  evidence  of 
direct  performance  feedback  mechanisms  in  place.  For  this  alternative,  insights  will  be 
offered  into  why  an  SQC  plan  is  important,  how  it  affects  system  performance,  and  what 
the  SQC  plan  should  address.  Additional  information  on  statistical  quality  control  is 
provided  in  appendix  B. 

4-1.2  Assumptions.  The  assumptions  made  in  constructing  this  design  were: 

•  the  false  call  rate  for  a  human  inspector  is  2.5  percent  and 

•  the  probability  of  detecting  a  crack  for  a  human  inspector  is  90  percent. 

The  assumptions  regarding  the  false  call  rate  and  the  probability  of  detecting  a  crack 
are  based  on  analysis  detailed  in  appendix  D. 

4-1.3  Configuration.  The  design  for  alternative  1  is  composed  of  the  same  com¬ 
ponents  as  the  baseline  without  the  FPIM,  and  the  addition  of  a  blade  counter  on  the 
AFPPM  to  increase  the  accuracy  of  the  system  statistics.  Observations  of  the  current  in¬ 
spection  process  indicate  that  an  accurate  count  of  the  number  of  blades  actually  processed 
through  the  inspection  system  is  not  maintained.  Not  knowing  the  total  number  of  blades 
entering  the  system  can  lead  to  inaccurate  performance  statistics.  Presently,  blade  count 
data  are  entered  directly  from  accompanying  paperwork  without  confirmation.  In  order  to 
maintain  usable  statistics  on  the  system’s  performance,  the  number  of  blades  entering  the 
inspection  system  must  be  accurate.  One  way  to  solve  this  problem  without  overburden¬ 
ing  the  workers  is  to  add  an  automated  blade  counter  to  the  system.  Before  the  basket  of 
blades  is  processed  by  the  AFP!  M,  it  will  be  placed  on  an  automated  blade  counter  that 
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can  accurately  count  and  record  the  number  and  type  of  blades  entering  the  system.  See 
appendix  B  for  a  description  of  the  blade  counter.  Figure  4.1  shows  the  blade  flow  for  this 
design. 


Figure  4.1.  Alternative  1. 


4- 1-4  Statistical  Quality  Control  Plan.  The  current  system  operates  in  an  uncon¬ 
trolled  manner.  The  critical  performance  variables  are  neither  identified  nor  adequately 
monitored.  Therefore,  the  state  of  control  for  the  IBIS  process  is  unknown  and  the  process 
performance  is  only  estimated.  A  coherent  SQC  plan  will  help  to  monitor  the  system  and 
provide  a  way  to  continually  improve  the  system’s  performance.  The  purpose  of  the  SQC 
plan  is 

•  quantify  the  IBIS  performance 

•  feedback  performance  information  to  the  system  by  identifying  unacceptable  process 
performance  and  failure  modes  which  will  allow  immediate  correction  of  deficiencies 

•  provide  a  method  to  identify  areas  of  continued  improvement 

The  SQC  plan  is  composed  of  three  sections:  random  sampling,  calibration  experi¬ 
ments,  and  design  of  experiments  methods.  Random  sampling  provides  a  means  to  gen¬ 
erate  an  understanding  of  the  IBIS  performance  by  examining  its  product,  inspected  jet 
engine  blades.  The  calibration  experiments  will  determine  whether  the  system  meets  per¬ 
formance  requirements.  The  design  of  experiments  will  identify  significant  variables  in  the 
blade  preparation  and  inspection  processes  whose  variability  must  be  closely  controlled. 
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Together,  these  three  parts  of  the  SQC  plan  will  help  the  engin^rs  maintain  process 
control,  insure  acceptable  system  performance,  identify  problems,  and  implement  system 
improvements. 

The  entire  SQC  concept  revolves  around  two  critical  issues:  system  performance  and 
process  control  (17).  The  system  performance  must  insure  a  product  that  meets  stipulated 
design  requirements.  For  the  IBIS  these  requirements  must  be  established  by  the  jet  engine 
design  engineers,  in  consultation  with  the  NDI  engineers.  The  jet  engine  safety  must  be 
insured  from  the  design  perspective,  but  the  inspection  system  must  be  capable  of  attaining 
the  established  goals. 

Process  control  must  insure  the  system  output  variation  falls  within  acceptable  pa¬ 
rameters.  If  the  system  output,  on  average,  is  acceptable,  yet  the  fluctuation  of  the  output 
over  time  is  large,  or  if  the  variation  is  not  independent  of  time,  the  system  is  operating 
out  of  control.  The  IBIS  must  insure  both  process  control  and  acceptable  performance  if 
it  is  to  represent  a  quality  system.  The  three  parts  of  the  SQC  plan,  random  sampling, 
calibration  experiments,  and  design  of  experiments  will  work  together  to  insure  acceptable 
overall  system  performance  and  consistent  process  control. 

4.1.4-1  Random  Sampling.  Currently,  accurate  performance  statistics  for  the 
IBIS  are  not  maintained  which  limits  estimations  of  system  performance  to  guess  work.  The 
random  sampling  of  inspected  engine  blades  will  allow  the  calculation  of  two  potentially 
useful  performance  indicators;  the  percentage  of  missed  detections  and  the  percentage  of 
false  calls  for  the  system.  A  missed  detection  occurs  when  a  cracked  blade  is  not  correctly 
identified  and  is  returned  to  service.  A  false  call  occurs  when  a  blade  that  has  no  cracks  is 
identified  as  cracked  and  discarded  as  scrap  (see  appendix  B).  The  more  significant  of  the 
two  errors  is  a  missed  detection  due  to  the  ramifications  of  putting  a  cracked  blade  back 
into  service  (see  appendix  D). 

The  random  sampling  portion  of  the  SQC  plan  has  three  inspection  schemes.  The  first 
inspection  scheme  calls  for  a  comprehensive  sampling  plan  of  one  or  two  fracture  critical 
blade  types.  MIL-STD-105E  defines  the  sample  plan  to  be  used  for  the  selected  blade  types. 
The  blades  identified  as  acceptable  are  randomly  sampled  according  to  the  sampling  plan. 
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These  sampled  blades  are  then  inspected,  which  will  determine  the  acceptable  quality  level 
as  defined  in  MIL-STD-105E  (appendix  B).  This  information  provides  a  means  of  recording 
the  system  performance.  The  SQC  plan  can  identify  when  the  system  performance  is 
unacceptable,  that  is,  when  the  system  fails.  Once  these  system  failures  emerge,  the 
NDI  engineers  must  identify  the  causes  and  fix  the  problems.  If  the  system  failures  are 
not  readily  identified,  then  the  probability  of  jet  engine  problems  related  to  turbine  and 
compressor  blade  failures  increase. 

An  important  assumption  in  this  scheme  lies  in  the  proposition  that  the  few  blade 
types  chosen  for  the  sampling  represent  all  the  blade  types  inspected.  This  assumption 
may  be  reasonable,  but  the  extrapolation  of  information  for  all  the  blades  from  a  limited 
portion  of  them  presents  the  potential  to  optimize  the  system  for  a  characteristic  peculiar 
only  to  the  sampled  blade  types.  In  order  to  protect  against  this  danger,  the  SQC  plan 
relies  on  a  second  random  sampling  scheme.  This  scheme  calls  for  a  random  sampling  of 
all  outgoing  blades.  Under  this  scheme  the  NDI  engineers  randomly  sample  all  the  blades 
identified  as  acceptable  on  a  periodic  basis,  say  two  or  three  times  a  week.  These  blades 
are  inspected,  and  the  product  quality  is  determined  (see  appendix  B). 

The  third  random  sampling  scheme  calls  for  the  inspection  of  a  sizable  portion  of  the 
blades  identified  as  unacceptable.  This  scheme  serves  a  dual  purpose.  First,  it  determines 
the  percentage  of  blades  that  are  actually  cracked  out  of  all  the  blades  rejected  by  the 
system.  This  percentage  must  fall  within  acceptable  limits  or  the  SQC  plan  will  call  for 
corrective  actions.  Second,  the  blades  correctly  identified  as  cracked  can  be  characterized 
and  used  as  a  reference  standard  in  the  calibration  experiments,  which  is  the  second  part 
of  the  SQC  plan  (see  appendix  B). 

In  order  to  perform  the  random  sampling  schemes,  inspection  techniques  that  can 
reliably  identify  cracks  and  accurately  measure  their  size  must  be  identified.  An  inspection 
technique  that  provides  a  high  rate  of  confidence  for  the  identification  of  cracks  involves 
extensive  cleaning  of  the  blades  and  using  a  fluorescence  microscope  to  view  the  blades. 
The  fluorescence  microscope  is  also  capable  of  accurately  measuring  the  length  of  cracks 
found  on  the  blades. 
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4.1. 4-2  Calibration  Experiments.  The  calibration  experiments  provide  a  re¬ 
peatable  means  to  calibrate  the  IBIS  and  determine  if  the  system  is  appropriately  tuned. 
These  experiments  can  also  be  used  to  generate  the  system  Probability  of  Detection  (POD) 
curves  which  provide  a  performance  measure  that  can  be  used  to  evaluate  the  system.  Rum- 
mel  presents  two  standard  tools  that  lend  themselves  to  this  system  calibration  concept: 
qualification  standards  and  reference  standards  as  a  means  of  controlling  the  NDI  pro¬ 
cess  (51).  The  qualification  standard  represents  the  means  to  perform  the  POD  analysis, 
thereby  establishing  the  system  performance.  The  IBIS  qualification  standard  is  a  set  of 
blades  with  artificially  grown  cracks.  The  length  of  these  cracks  are  accurately  measured 
and  include  cracks  that  represent  the  smallest  crack  the  IBIS  must  consistently  detect. 
The  reference  standard  represents  a  means  to  frequently  gauge  the  performance  capabili¬ 
ties  of  the  system  by  periodically  inserting  the  reference  standard  blades  into  the  system 
observing  the  results. 

4. 1.4.3  Design  Of  Experiments.  Performing  a  designed  experiment  on  a  sys¬ 
tem  provides  information  about  the  present  system’s  performance  and  identifies  variables 
that  drive  the  system’s  response.  Knowledge  of  the  present  system  performance  allows 
system  managers  to  insure  their  product  is  leaving  with  the  best  quality  possible.  Identi¬ 
fication  of  significant  variables  allow  managers  to  make  informed  decisions  on  the  need  to 
commit  additional  resources  to  control  specific  variables.  For  the  IBIS,  the  product  can  be 
represented  by  the  blades  leaving  the  system,  and  a  measure  of  quality  ^^s  the  number  of 
blades  incorrectly  classified. 

As  mentioned  in  chapter  II,  there  is  a  cost  to  the  system  for  incorrectly  classifying 
blades.  This  cost  is  driven  by  the  probability  of  detecting  a  bad  blade  and  the  false  call 
rate.  These  criteria  are  a  natural  measure  of  system  quality.  For  example,  to  improve  the 
system  it  is  necessary  to  increase  the  probability  of  detecting  a  bad  blade  and  decrease  the 
probability  of  calling  good  blades  bad.  Presently,  data  on  the  probability  of  detection  and 
false  call  rate  for  the  IBIS  are  not  available,  therefore  a  quantitative  measure  of  outgoing 
blade  quality  as  mentioned  above  is  not  possible.  This  lack  of  information  poses  two 
problems.  First,  it  is  difficult  to  compare  alternative  solutions  when  the  baseline  is  not 
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quantified.  Second,  SA-ALC  desires  to  expand  their  blade  inspection  capability,  possibly 
to  include  work  from  the  civilian  sector.  In  order  to  attract  additional  customers,  IBIS 
personnel  need  to  have  their  system  quantified  in  enough  detail  to  convince  customers  that 
it  is  a  practical  solution  to  the  customers’  blade  inspection  needs. 

4.1-5  Expected  System  Performance.  The  human  inspectors  are  capable  of  inspect¬ 
ing  on  average  at  least  1  blade  every  5  seconds,  if  its  a  good  blade  and  every  30  seconds,  if 
its  a  bad  blade.  The  average  human  inspector’s  false  call  rate  is  approximately  2.5  percent 
with  a  probability  of  identifying  a  crack  of  approximately  90  percent  (see  appendix  D).  A 
human  inspector  can  consistently  identify  cracks  .070"  long  with  a  lower  limit  around  .030" 
according  to  Ward  Rummel  of  Martin  Marietta  who  has  performed  extensive  research  in 
this  area  (50,  51).  Table  4.1  summarizes  the  components  and  expected  performance  of 
alternative  1. 


Component 

Quantity 

AFPPM 

1 

ABR 

-  Manipulator  Sys 

0 

-  CCD  w/light  source 

0 

-  Master  Computer 

0 

Human  Inspectors 

3 

SQC 

-  Inspectors 

1 

-  Microscope  and  Blade  counter 

1 

Performance 

-  False  call  rate 

2.5  percent 

-  Avg  Inspection  Time 

6.25  sec/ blade 

-  Probability  ID  Crack 

90  percent 

Table  4.1.  Summary  of  alternative  1. 


4-1.6  Summary.  Alternative  1  attempts  to  solve  the  problems  of  the  current  system 
by  removing  the  cause  of  the  majority  of  the  system’s  problems  (the  FPIM)  and  providing 
a  means  to  more  closely  monitor  and  control  the  inspection  process  through  an  SQC  plan. 
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This  alternative  offers  a  low  capital  cost,  short  term  solution  for  the  current  system,  but 
fails  to  adequately  address  the  issue  of  inconsistent  human  inspection. 

4.2  Alternative  2:  Semi-Automated  Inspection 

4.2.1  Overview.  The  design  of  alternative  2  was  motivated  by  the  need  to  improve 
upon  the  problem  areas  of  the  current  system.  Significant  problem  areas  of  the  current 
system  addressed  with  this  alternative  are  poor  maintainability,  high  false  call  rate,  difficult 
scan  plan  generation,  and  lack  of  flexibility.  Alternative  2  is  optimized  to  increase  the 
probability  of  identifying  a  crack  and  reduce  the  false  call  rate  while  maintaining  human 
inspectors  in  the  system.  The  reduction  in  error  rates  is  achieved  by  using  a  neural  network 
based  classification  algorithm  for  the  automated  blade  reader  (ABR)  (see  appendix  G). 

Alternative  2  is  a  semi-automated  inspection  system  that  uses  an  automated  blade 
reader(ABR)  and  human  inspectors  to  inspect  blades  for  cracks.  The  ABR  is  comprised 
of  readily  available,  industry  standard  components  for  blade  manipulation  and  sensoring 
which  contributes  to  high  system  maintainability.  The  sensor  is  a  charge  coupled  device 
(CCD)  camera  which  decreases  data  collection  time  by  eliminating  the  need  for  elaborate 
scan  plans.  Classification  software  relies  on  a  neural  network  to  provide  rapid  and  accu¬ 
rate  blade  classification.  Adding  automated  inspection  and  an,  effective  statistical  quality 
control  (SQC)  plan  will  provide  a  more  consistent  inspection  process. 

Alternative  2  is  best  summarized  as  replacing  the  present  FPIM  with  a  new  ABR 
and  having  humans  inspect  the  suspect  blades.  The  ABR  operates  similar  to  the  current 
FPIM  and  maintains  humans  inspectors  within  the  inspection  process.  The  design  was 
purposely  developed  in  this  manner  to  answer  user  acceptance  and  confidence  problems. 
Since  the  design  is  familiar  and  relies  on  proven  components,  users  should  be  more  willing 
to  accept  this  design  and  more  confident  of  its  performance  (1). 

4.2.2  Assumptions.  The  assumptions  made  in  constructing  this  design  were: 


•  the  blades  will  be  loaded  in  the  preprocessing  basket  the  same  way  every  time, 

•  the  maximum  blade  length  is  10  inches. 
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•  the  maximum  blade  weight  is  16  ounces, 

•  the  preprocessing  baskets  will  remain  unchanged, 

•  false  call  rate  for  the  ABR  is  20  percent,  and 

•  the  ABR  probability  of  detecting  a  crack  is  99  percent. 

The  assumptions  regarding  the  false  call  rate  and  the  probability  of  detecting  a  crack 
are  based  on  analysis  detailed  in  appendices  D  and  G. 

4-2.3  Configuration.  Alternative  design  2  is  composed  of  the  AFPPM  with  blade 
counter,  an  ABR,  and  at  least  one  human  inspector.  All  blades  will  first  be  inspected 
by  the  automated  blade  reader  (ABR)  which  will  classify  the  blades  as  good,  cracked, 
or  suspect.  Only  those  blades  identified  as  suspect  will  be  sent  to  the  human  inspectors 
for  inspection  and  final  disposition.  The  major  difference  between  this  design  and  the 
current  system  is  that  the  ABR  screens  all  blades  instead  of  acting  in  parallel  with  manual 
inspection  stations.  Figure  4.2  shows  the  blade  flow  of  this  design. 

Initial  attempts  to  solve  the  problems  of  the  present  system  involved  analyzing  the 
present  configuration  and  identifying  upgrades  to  the  troublesome  areas.  Further  research 
revealed  that  the  system  was  so  interconnected  and  dependent  on  the  computer  system  that 
simple  component  replacement  was  not  feasible  without  a  dramatic  redesign.  Additionally, 
the  entire  FPIM  is  composed  of  prototypes  components  making  it  extremely  difficult  to 
make  component  upgrades.  The  few  changes  that  could  have  been  made  would  have 
necessitated  large  amounts  of  work  for  relatively  little,  if  any,  performance  gains.  The  idea 
of  making  modular  upgrades  to  the  FPIM  was  eliminated  for  these  reaisons. 

This  alternative  attempts  to  remedy  the  deficiencies  of  the  current  system  by  using 
the  same  design  concept  as  the  baseline  and  modifying  or  replacing  subsystem  components. 
Alternative  2  does  not  attempt  to  offer  a  complete  solution  for  the  high  false  call  rate 
directly.  Rather,  its  aim  is  to  reduce  the  false  call  rate  to  an  acceptable  leVel  for  the 
automated  inspection  by  optimizing  the  preprocessing  procedure  and  by  using  a  more 
sophisticated  indication  classification  algorithm. 

The  primary  objective  of  this  design  is  to  reduce  the  amount  of  human  inspection  by 
rapidly  screening  out  the  good  and  cracked  blades,  leaving  the  suspect  blades  for  human 
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inspectors.  The  ABR  should  relieve  human  inspectors  of  inspecting  blades  that  have  no 
relevant  indications.  Since  the  labeled  good  and  cracked  blades  constitute  approximately 
80  percent  of  the  total  blades  inspected,  the  system  has  the  potential  to  dramatically  reduce 
the  humaji  inspectors’  workload  to  approximately  20  percent  of  all  blades.  The  emphasis  is 
to  reduce  the  workload  of  the  human  inspectors  in  hopes  of  improving  their  performance  as 
well  as  increasing  the  use  of  automation  in  the  inspection  process  to  enhance  consistency 
and  quality.  The  automated  system  will  more  consistently  and  accurately  identify  the 
obviously  good  and  cracked  blades  while  the  human  inspectors  will  use  their  superior 
pattern  recognition  skills  to  classify  the  more  difficult  blades.  This  alternative  design  seeks 
to  exploit  the  advantages  that  automation  allows  while  retaining  the  human  inspectors  to 
compensate  for  the  potential  disadvantages  of  a  fully  automated  inspection  system. 


Good  Blades 
Bad  Blades 


Good  Blades  Bsd  Blades 


Figure  4.2.  Alternative  2. 


4-2.4  System  Components.  The  major  components  of  alternative  2  are  the  AFPPM 
with  blade  counter,  the  ABR,  at  least  one  human  inspector,  and  the  SQC  plan  developed 
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in  alternative  1.  The  human  inspectors  will  perform  essentially  the  same  tasks  as  in 
alternative  1  with  the  exception  lying  in  the  composition  of  their  inspection  items  (all 
suspect  blades).  The  main  difference  is  in  the  ABR  and  the  computer  system.  The  ABR  is 
an  FP  visual  inspection  system  that  consists  of  a  standard  industrial  robotic  manipulator,  a 
charge  coupled  device  (CCD)  camera,  an  ultraviolet  light  source,  and  a  blade  classification 
algorithm  interconnected  and  controlled  by  the  master  computer  system.  Due  to  the  use 
of  the  ultraviolet  light,  the  sensitivity  of  the  CCD  camera  to  the  effects  of  visible  light 
on  image  collection,  and  safety  concerns  of  using  an  industrial  manipulator,  the  ABR 
will  require  an  inspection  booth  similar  to  those  used  by  the  human  inspectors  with  the 
addition  of  a  barrier  as  a  standard  safety  precaution. 

Alternative  2  eliminates  the  need  to  manually  load  and  unload  the  manipulator.  This 
helps  to  decrease  the  amount  of  time  required  to  inspect  a  blade  and  eliminates  the  need  to 
have  a  human  constantly  monitor  the  system.  It  also  simplifies  the  scan  plans  and  reduces 
the  data  collection  time  since  the  CCD  can  image  an  entire  blade  with  two  exposures 
rather  than  having  to  scan  the  entire  blade  surface  pixel  by  pixel.  The  advantages  offered 
by  the  CCD  camera  allows  the  ABR  to  inspect  one  blade  every  10  seconds  on  average 
(see  appendix  F).  Alternative  2  also  uses  a  blade  classification  algorithm  based  on  a 
trained  neural  network  rather  than  a  limited  rule  based  decision  algorithm.  The  neural 
network  relies  on  more  indication  features  to  classify  blades  which  enables  the  system  to 
more  accurately  classify  blades.  This  enhanced  classification  ability  will  enable  the  ABR 
system  to  attain  a  probability  of  detection  of  99  percent  and  a  false  call  rate  of  25  percent. 
Appendix  G  provides  support  for  the  classification  performance  estimates.  The  baseline 
system  classifies  blades  into  three  separate  categories,  but  due  to  its  high  false  call  rate,  the 
inspectors  lump  the  labeled  cracked  and  suspect  blades  together  as  suspect.  By  sending 
blades  that  have  been  labeled  cracked  by  the  FPIM  to  the  human  inspectors,  the  inspectors 
are  provided  an  opportunity  to  misclassify  a  cracked  blade.  This  increases  the  probability 
of  putting  a  cracked  blade  back  into  service  (see  appendix  D). 

4-2. 4-1  Robotic  Manipulator.  A  consideration  of  this  design  is  the  expected 
increase  in  both  the  number  and  types  of  blades  to  be  inspected.  The  current  system  cannot 
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meet  the  projected  increases  in  either  case  due  to  its  configuration  and  the  complexity  of 
its  blade  scan  plans.  Additional  flexibility  is  provided  by  replacing  the  current  manipulator 
with  a  more  flexible  and  maintainable  manipulator  which  will  allow  the  system  to  expand 
its  range  of  blade  types  it  can  inspect.  Replacing  the  current  manipulator  will  also  do  away 
with  the  requirement  to  manually  load  and  unload  the  ABR  because  the  new  manipulator 
will  pick  and  place  the  blades  automatically  from  the  preprocessing  baskets.  Replacing  the 
laser  scanning  sensor  with  a  CCD  camera  will  negate  the  need  for  extensive  scan  plans  and 
reduce  the  time  required  for  data  acquisition.  This  alternative  design  will  operate  much 
faster  and  be  more  flexible  to  adapt  to  changing  requirements. 

Initial  analysis  of  the  task  indicates  that  the  ideal  manipulator  for  this  design  is  a  Se¬ 
lective  Compliance  Assembly  Robot  Arm  (SCARA)  type  manipulator.  The  manipulation 
task  requires  high  velocity,  long  distance  movements  in  the  horizontal  plane  while  moving 
between  the  two  baskets;  and  high  velocity,  short  movements  in  the  vertical  plane  when  re¬ 
trieving  blades  from,  or  placing  them  into  the  baskets.  The  major  disadvantage  arises  from 
the  typically  short  vertical  travel  limits  of  SCARA  manipulators  suitable  for  the  task.  An 
average  vertical  travel  limit  is  approximately  12  inches  which  severely  restricts  the  length 
of  blades  that  the  system  is  capable  of  inspecting  (see  appendix  F). 

A  vertically  articulated  manipulator  is  significantly  more  flexible  allowing  a  much 
greater  range  of  blade  sizes  and  shapes  to  be  inspected.  The  major  potential  disadvantage 
when  compared  to  a  SCARA  manipulator  is  that  a  vertically  articulated  design  is  usually 
slower  moving  laterally.  However,  the  manipulator  that  was  chosen  compares  favorably 
given  that  a  typical  SCARA  design’s  maximum  joint  speeds  are  approximately  300  degrees 
per  second  while  the  articulated  design’s  are  240  degrees  per  second.  The  trade  ofl!"  in 
selecting  the  vertically  articulated  design  was  a  slight  speed  disadvantage  during  lateral 
motion  for  more  flexiblility  for  future  upgrades. 

Another  tradeoff  occurs  based  on  the  system  constraint  that  industry  standard  com¬ 
ponents  be  used.  By  using  an  industry  standard  manipulator,  the  surge  capacity  of  the 
system  wa.s  degraded  (see  throughput  analysis  in  section  5.6).  The  robotic  simulation 
model  described  in  appendix  F  shows  that  industry  standard  robotic  manipulators  are 
capable  of  manipulating  a  blade  for  imaging  and  disposition  in  7  seconds.  It  is  probably 


4-12 


possible  to  perform  the  manipulation  task  in  less  time  with  a  custom  manipulator,  but 
the  advantages  of  increased  maintainability  and  upgradability  offered  by  using  standard 
components  outweigh  surge  capacity  considerations.  Consequently,  the  ABR  must  be  op¬ 
erated  for  approximately  13  hours  per  day  to  inspect  1.2  million  blades  per  year.  If  one 
8  hour  shift  per  day  of  operation  or  increased  surge  capacity  is  desired,  then  additional 
ABRs  must  be  considered. 

The  robotic  manipulator  is  a  standard  industrial  manipulator  that  is  commercially 
available.  As  one  example,  the  manipulator  manufactured  by  Motoman,  model  K6SB,  is 
representative  of  the  type  of  industrial  robots  that  are  widely  available  and  capable  of 
performing  the  required  manipulation.  This  manipulator  is  a  vertically  articulated  design 
with  3  revolute  joints  in  the  arm  and  3  revolute  joints  in  the  wrist.  The  manipulator 
has  6  degrees-of-freedom  to  allow  it  to  reach  in  and  around  the  preprocessing  baskets  to 
extract  and  insert  the  blades.  The  flexibility  of  this  articulated  design  allows  the  current 
basket  and  conveyor  design  to  remain  mostly  unchanged.  The  end-effector  is  a  simple 
parallel  jaw  gripper  with  tactile  sensors  located  on  the  grasping  surface  to  indicate  proper 
dovetail  grasp.  Additional  conveying  systems  will  be  added  to  transport  the  baskets  of 
blades  from  the  AFPPM  to  the  manipulator,  and  from  the  ABR  to  the  human  inspectors 
and  to  postprocessing.  Some  of  the  on-hand  conveyors  or  systems  similar  to  the  ones  used 
currently  may  be  used  to  perform  these  tasks. 

The  manipulator  will  retrieve  an  individual  blade  from  its  preprocessing  basket, 
present  one  side  to  the  CCD  camera  for  imaging,  rotate  the  blade  180  degrees  to  present 
the  opposite  side  of  the  blade,  and  then  place  the  blade  either  back  in  the  basket  if  it  is 
identified  as  good,  in  a  blade  scrap  receptacle  if  it  is  identified  as  cracked,  or  in  another 
basket  that  will  go  to  the  human  inspectors  if  the  blade  is  identified  as  suspect.  The 
manipulator  will  continue  retrieving  and  disposing  of  the  blades  until  all  the  blades  in  the 
preprocessing  basket  are  inspected.  After  the  initial  screening  is  complete,  the  basket  of 
good  blades  will  proceed  to  postprocessing  while  the  basket  of  suspect  blades  will  proceed 
to  the  human  inspectors  for  more  detailed  inspection.  Figure  4.3  shows  the  configuration 
of  the  ABR  inspection  booth. 
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Figure  4.3.  Workcell  Layout 


Based  on  IGRIP  simulations  (see  appendix  F),  the  majority  of  the  inspection  time 
(approximately  7  seconds  per  blade)  is  used  to  manipulate  the  blades.  Recognizing  this  as  a 
limitation,  a  fairly  fast  robot  was  chosen.  The  maximum  joint  velocities  for  the  K6SB  range 
from  240  degrees  per  second  to  400  degrees  per  second.  We  also  minimized  the  distances 
the  manipulator  has  to  move  the  blades  in  order  to  reduce  the  required  manipulation  time. 

4-2. 4-2  Sensor.  The  sensor  subsystem  is  composed  of  a  light  source,  a  CCD 
camera,  and  associated  lenses  for  the  camera.  The  CCD  camera  should  be  an  area  scan 
camera  with  an  array  of  at  least  2048  x  2048  pixels  so  that  the  largest  blade  expected 
(10"  X  10")  can  be  inspected  with  only  one  image  per  side  while  maintaining  a  resolution 
of  .005"  per  pixel.  The  camera  specifications  are  detailed  in  appendix  J.  The  array  should 
be  composed  of  square  pixels  to  prevent  distortion  of  the  image  due  to  blade  surface 
orientation.  Filters  must  be  used  to  let  only  the  light  radiating  from  the  penetrant  reach 
the  camera.  Lens  selection  is  driven  by  the  distance  between  the  blade  surface  and  the 
camera  (see  appendix  J). 

For  optimum  performance,  the  light  source  must  produce  a  narrow  bandwidth  beam 
centered  on  365  nm  and  of  sufficient  power  to  ensure  the  intensity  of  the  light  will  be 
at  least  800  microwatts  per  square  centimeter  at  the  blade  surface.  The  desired  light 
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properties  can  be  obtained  by  using  a  less  optimum  light  source  and  filtering  the  light  to 
obtain  the  desired  center  wavelength.  However,  care  must  be  exercised  to  ensure  that  the 
required  intensity  at  the  blade  surface  is  maintained  (see  appendix  J). 

4-2. 4. 3  Blade  Classification  Algorithm.  The  blade  classification  algorithm  is 
capable  of  classifying  indications  as  either  good,  cracked  or  suspect.  It  cannot  further 
classify  the  blades  as  dented,  scratched,  or  pitted  because  the  system  must  operate  at 
a  high  rate  of  speed.  The  main  concern  is  to  quickly  process  a  large  number  of  blades 
and  to  correctly  classify  the  majority  of  the  good  blades.  By  identifying  the  clearly  good 
and  bad  blades  and  removing  them  from  the  set  to  be  inspected  manually,  the  human 
inspector’s  workload  is  reduced  to  approximately  20  percent  of  the  total  blades  inspected. 
The  advantage  gained  from  quickly  processing  a  large  number  of  blades  through  the  system 
will  significantly  outweigh  the  disadvantage  associated  with  labeling  a  portion  of  the  blades 
as  suspect. 

The  software  has  two  major  components:  image  processing  and  classification.  The 
image  processing  portion  of  the  software  extracts  a  variety  of  features  from  the  raw  image 
data.  The  features  can  then  be  used  to  identify  indications  associated  with  defects.  Once 
the  features  are  collected,  the  classification  software  can  determine  if  the  indications  are 
significant  enough  to  call  the  blade  suspect.  Of  the  many  methods  that  could  be  used,  a 
multilayer  perceptron  neural  network  was  evaluated  for  this  decision  analysis  task  because 
it  operates  very  quickly  and  has  low  memory  requirements.  The  network  must  be  trained 
with  data  representative  of  what  the  system  would  encounter  during  normal  operation.  The 
output  of  the  network  would  be  interpreted  to  minimize  the  number  of  defective  blades 
called  good  while  maintaining  the  number  of  good  blades  called  bad  below  20  percent. 

Appendix  G  provides  a  detailed  description  of  the  type  of  image  processing  and  neural 
network  software  required  for  alternative  2.  Additionally,  it  contains  the  results  from  an 
example  neural  network  that  used  data  collected  from  the  current  FPIM.  The  developed 
software  demonstrates  some  of  the  advantages  of  using  advanced  processing  techniques  to 
collect  features  and  to  classify  blades.  It  also  demonstrates  the  feasibility  of  using  neural 
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networks  to  classify  blades  and  the  potential  to  improve  the  classification  ability  of  an 
automated  system. 

4.2. 4-4  Master  Computer  Subsystem.  An  industry  standard  VME  bus  com¬ 
puter  was  chosen  for  the  master  computer  subsystem.  The  computer  subsystem  serves  as 
the  central  control  of  the  ABR.  The  computer  will  initiate  and  monitor  the  manipulation 
of  the  engine  blades.  In  addition,  the  computer  subsystem  must  coordinate  the  sensor 
and  manipulator  so  the  blade  surface  can  be  accurately  imaged.  The  sensor  subsystem 
provides  the  computer  with  the  available  blade  data.  The  computer  m’lst  process  this 
data  and  classify  the  jet  engine  blade  under  inspection.  The  computer  is  partitioned  into 
three  functional  units:  data  acquisition  «e''tion,  data  processing  and  storage  section,  and 
user  interface  section. 

A  frame  grabber  and  an  image  processor  comprise  the  data  acquisition  section.  The 
frame  grabber  functions  as  the  bridge  between  the  sensor  and  the  computer  system.  The 
image  processor  accepts  the  data  from  the  frame  grabber  and  performs  preliminary  process¬ 
ing.  This  processing  prepares  the  data  for  the  classification  algorithm.  The  frame  grabber 
and  the  image  processor  are  directly  connected  via  a  VME  system  bus,  (VSB),  which  op¬ 
erates  at  50Mbytes  per  second.  The  use  of  this  direct  connection  frees  the  backplane  for 
other  use. 

The  data  processing  and  storage  section  of  the  computer  accepts  the  preprocessed 
data  from  the  image  processor  via  the  VME  backplane.  Two  Motorola  68040  based  single 
board  computers,  a  Small  Computer  System  Interface  (SCSI)  mass  storage  device  with 
a  tape  drive  storage  media,  additional  Dynamic  Random  Access  Memory  (DRAM)  as 
necessary,  and  a  chassis  with  any  required  components  to  operate  the  system  comprise 
this  section  of  the  computer  subsystem.  One  of  the  single  board  computers  will  accept 
the  data  from  the  image  processor,  process  the  data,  and  classify  the  blade.  The  final 
classification  will  pass  directly  to  the  manipulation  subsystem  via  a  serial  interface.  Any 
relevant  data  will  be  written  to  the  SCSI  mass  storage  system.  This  single  board  computer 
will  also  have  an  Ethernet  connection  to  the  system  user  interface.  The  second  single  board 
computer  will  coordinate  the  manipulator  and  sensor  subsystem  via  serial  interfaces. 
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The  user  interface  section  is  a  Unix  workstation  that  will  be  used  to  access  the  VME 
computer.  IBIS  engineering  personnel  will  also  use  this  interface  section  to  perform  any 
relevant  computing  functions.  Therefore,  the  workstation  must  have  access  to  the  VME 
mass  storage  and  significant  processing  capabilities.  A  Sun  Sparcstation  2  is  appropriate 
for  this  section  of  the  computer  subsystem. 

A  VME  bus  computer  system  was  chosen  for  three  primary  reasons.  First,  VME 
bus  systems  represent  the  industry  standard  for  real  time,  industrial  computing.  Second, 
the  system  is  inherently  upgradable.  When  new  requirements  are  identified  or  when  new 
capabilities  emerge,  they  can  be  implemented  in  the  existing  system  as  opposed  to  creating 
an  entirely  new  system.  Finally,  the  VME  system  is  designed  for  multi-microprocessor 
capability  which  allows  parallel  processing.  The  ability  to  parallel  process  enables  the 
computer  system  to  perform  data  acquisition,  image  processing  and  storage,  and  system 
interface  functions  quickly. 

4.2.5  Expected  System  Performance.  The  ABR  system  should  be  capable  of  in¬ 
specting  on  average  at  least  1  blade  every  10  seconds  (3  seconds  for  data  collection  and 
processing  and  7  seconds  for  blade  manipulation)  with  a  false  call  rate  no  higher  than  20 
percent  (see  appendices  G  and  F)  and  a  probability  of  detecting  a  crack  of  99  percent  (see 
appendix  D).  The  estimate  for  data  collection  and  image  processing  time  was  derived  from 
computer  vendors’  estimates  based  on  a  computer  system  similar  to  the  one  described  in 
section  4.2.4.4.  These  estimates  are  consistent  with  performance  figures  for  similar  inspec¬ 
tion  systems  (1,  51).  Combining  the  ABR’s  and  the  human  inspectors  performance,  give 
the  total  system  a  5.5  percent  false  call  rate,  a  99.5  percent  probability  of  detecting  a 
cracked  blade,  and  an  average  inspection  time  of  11  seconds  per  blade  (see  table  5.20  for 
throughput  values). 

Table  4.2  summarizes  the  components  and  expected  performance  of  alternative  2. 

4.2.6  Models.  The  robotics  simulation  program,  IGRIP,  developed  by  Deneb  Robotics, 
Inc.  was  used  to  simulate  the  ABR.  The  simulation  provided  a  means  to  evaluate  the  design 
for  feasibility  of  workcell  layout  as  well  as  predict  the  time  required  to  inspect  a  basket  of 
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Component 

Quantity 

AFPPM 

1 

ABR 

-  Manipulator  Sys 

1 

-  CCD  w/light  source 

1 

-  Master  Computer 

1 

Human  Inspectors 

1 

SQC 

-  Inspectors 

1 

-  Microscope  and  Blade  counter 

1 

Performance 

-  False  call  rate 

5.5  percent 

-  Avg  Inspection  Time 

11  sec/ blade 

-  Probability  ID  Crack 

99.5  percent 

Table  4.2.  Summary  of  alternative  2. 


blades  for  the  ABR.  The  workcell  layout,  the  camera  exposure  time,  the  false  call  rate,  and 
the  manipulator  specifications  were  varied  in  optimizing  the  design.  Appendix  F  provides 
a  more  detailed  explanation  of  the  robotic  simulation. 

Alternative  3:  Fully  Automated  Inspection 

4.3.1  Overview.  The  third  alternative  is  a  fully  automated  inspection  system  which 
eliminates  the  need  for  human  inspectors  in  the  blade  inspection  process.  This  alternative 
is  an  elaboration  of  alternative  2  and  shares  its  basic  design  philosophy.  The  difference 
is  that  this  alternative  is  optimized  to  produce  a  reduced  false  call  rate  and  an  increased 
probability  of  crack  detection  using  a  multiple  CCD  image  subtraction  technique  (see 
appendix  C). 

The  primary  focus  of  this  design  is  to  collect  additional  useful  crack  features  from 
the  blades  to  enhance  the  ability  of  the  system  to  classify  blades.  A  significant  source  of 
false  calls  results  from  indications  caused  by  surface  contamination  on  the  blades.  These 
contaminants  absorb  the  penetrant  and  frequently  produce  indications  which  are  similar 
to  those  of  cracks  or  may  cover  a  large  enough  area  of  the  blade  to  effectively  conceal  a 
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crack.  Research  indicates  that  there  may  be  several  ways  to  collect  additional  features  that 
can  be  used  to  more  accurately  identify  cracked  blades.  Experimental  data  indicate  that 
taking  multiple  images  of  the  same  blade  at  different  times  and  subtracting  the  images, 
allows  one  to  effectively  eliminate  nonrelevant  indications  from  the  decision  image,  making 
it  easier  to  identify  a  crack.  The  data  indicate  two  images,  one  taken  before  applying 
developer  and  one  taken  after  developer  application,  are  sufficient  to  accurately  classify 
cracked  blades  (see  appendix  C). 

4-3.2  Assumptions.  The  assumptions  made  in  constructing  this  design  were: 


•  the  blades  will  be  loaded  in  the  preprocessing  basket  the  same  way  every  time, 

•  the  maximum  blade  length  is  10  inches, 

•  the  maximum  blade  weight  is  16  ounces, 

•  the  preprocessing  baskets  will  remain  unchanged, 

•  the  intelligent  automated  blade  reader  (lABR)  false  call  rate  is  2.5  percent,  and 

•  the  lABR  probability  of  detecting  a  crack  is  99.5  percent. 

The  assumptions  regarding  the  false  call  rate  and  the  probability  of  detecting  a  crack 
are  based  on  analysis  detailed  in  appendices  D  and  G. 

4.3.3  Configuration.  Alternative  3  consists  of  the  AFPPM,  an  intelligent  auto¬ 
mated  blade  reader  (lABR),  and  the  SQC  plan  developed  in  Alternative  1.  This  alter¬ 
native  does  not  require  human  inspectors,  since  the  lABR  is  fully  capable  of  classifying 
all  system  inspectable  blades.  The  primary  objective  of  this  design  is  to  remove  as  much 
of  the  variability  and  inconsistencies  in  the  inspection  process  through  the  use  of  a  fully 
automated  system.  The  design  emphasis  is  on  increasing  the  consistency  of  the  inspection 
process  through  automated  inspection  in  order  to  improve  the  qualtity  of  the  end  product. 

The  most  significant  feature  of  this  design  is  an  improved  classification  ability  made 
possible  through  the  collection  of  additional  features  from  blades  processed  with  FP.  In 
addition  to  using  the  same  features  as  in  alternative  2,  this  design  will  use  the  difference 
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in  bleedout  rate  of  contaminants  and  cracks  to  increase  the  accuracy  of  indication  classifi¬ 
cation  so  that  the  probability  of  identifying  a  crack  is  99.5  percent  and  the  false  call  rate 
is  2.5  percent. 

4.3.4  System  Components.  The  major  components  of  Alternative  3  are  the  AFPPM 
with  blade  counter  and  the  lABR.  Figure  4.4  shows  the  blade  flow  of  the  design.  The  main 
diflFerence  is  in  the  lABR.  The  LABR  is  an  FP  visual  inspection  system  that  consists  of  two 
standard  industrial  robotic  manipulators,  two  CCD  cameras  with  ultraviolet  light  sources, 
and  a  blade  classification  algorithm  interconnected  and  controlled  by  the  master  computer 
system. 


Figure  4.4.  Alternative  3. 


As  previously  noted,  the  designs  of  alternatives  2  and  3  are  very  similar.  Both  will  use 
similar  master  computer  system  hardware.  The  most  significant  differences  will  be  found  in 
the  lABR  and  the  AFPPM.  The  major  requirement  of  alternative  3  is  to  collect  additional 
features  from  similarly  processed  blades  in  order  to  improve  the  classification  ability  of  the 
system.  The  most  promising  method  of  collecting  additional  features  is  through  the  use 
of  multiple  images  of  a  blade  taken  at  different  times.  By  imaging  the  blade  at  different 
times,  before  and  after  developer  is  applied,  it  should  be  possible  to  distinguish  true  crack 
indications  from  indications  caused  by  something  other  than  cracks.  Since  the  blades  will 
be  imaged  before  the  developer  is  applied,  the  AFPPM  will  have  to  be  separated  into 
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two  sections  at  the  developer  application  module.  Currently  parts  other  than  blades  are 
processed  by  the  AFPPM  so  they  must  be  identified  as  they  enter  the  AFPPM  so  that  the 
manipulators  will  not  inspect  them.  Since  the  AFPPM  operator  must  enter  a  recipe  based 
on  the  type  of  parts  in  a  basket,  a  simple  message  to  the  master  computer  based  on  the 
operator’s  input  should  suffice.  Figure  4.5  shows  the  configuration  of  the  lABR  inspection 
booth. 


Figure  4.5.  Workcell  Layout 


The  robotic  manipulators,  CCD  camer<is,  light  sources,  and  master  computer  system 
are  the  same  as  those  used  for  alternative  2.  The  two  manipulators,  each  with  its  own  CCD 
camera  and  light  source,  are  arranged  at  separate  points  along  a  conveyor  system.  One 
manipulator  is  located  after  the  emulsifier  drying  module  and  the  other  is  located  after  the 
developer  blow-off  module.  Each  of  the  manipulators  perform  essentially  the  same  task 
as  in  alternative  2.  Immediately  after  the  basket  of  blades  exits  the  drying  module  after 
emulsification,  the  first  manipulator  will  retrieve  an  individual  blade  from  its  preprocessing 
basket,  present  one  side  to  the  CCD  camera  for  imaging,  rotate  the  blade  180  degrees  to 
present  the  opposite  side  of  the  blade,  and  then  place  the  imaged  blade  back  in  the  basket. 
The  manipulator  will  continue  this  process  until  all  the  blades  in  the  basket  are  imaged. 
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The  images  will  be  stored  on  the  master  computer  for  processing  after  the  blades 
are  imaged  a  second  time.  Once  all  the  blades  are  imaged  and  replaced  in  the  basket,  the 
basket  of  blades  will  continue  on  the  conveyor.  After  the  basket  of  blades  have  proceeded 
down  the  conveyor  line  for  a  short  dwell  time,  they  will  enter  the  developer  and  blowoff 
modules  for  further  processing.  After  the  specified  dwell  time,  the  second  manipulator 
will  process  the  blades  similar  to  the  first,  except  after  the  second  side  of  the  blade  is 
imaged,  the  manipulator  will  place  the  blade  either  back  in  the  basket  for  postprocessing 
if  it  is  identified  as  good  by  the  classification  algorithm,  or  place  it  in  another  basket  to  be 
discarded. 

The  robotic  simulation  data  reveals  that  the  initial  imaging  will  require  approxi¬ 
mately  5  minutes.  Since  the  AFPPM  is  capable  of  processing  one  basket  every  two  minutes, 
baskets  of  blades  processed  through  the  AFPPM  must  be  staggered  on  the  conveyor  so 
they  do  not  arrive  at  the  lABR  too  quickly  during  surges.  A  simple  way  to  avoid  creating 
a  queue  is  to  only  load  every  third  basket  receptacle  with  blades.  Another  way  would  be 
to  slow  the  conveyor  or  reduce  the  number  of  blades  per  basket.  Similar  surge  capacity 
concerns  addressed  under  alternative  2  also  apply  to  this  alternative. 

A  consideration  resulting  from  the  need  to  image  a  blade  more  than  once  is  the 
potential  need  to  present  the  blade  to  the  CCD  camera  in  such  a  way  to  allow  image 
subtraction.  In  order  for  image  subtraction  to  work  properly,  the  images  must  relate 
exactly.  The  images  can  be  aligned  mechanically  or  through  software  manipulation  of  the 
image  data.  Typical  manipulator  repeatability  is  on  the  order  of  0.001  to  0.004  inches 
which  helps  to  alleviate  blade  placement  concerns.  However,  there  remains  a  concern 
that  the  manipulator  will  not  grasp  the  blade  dovetail  in  exactly  the  same  location  or 
orientation  which  can  make  image  subtraction  difficult.  The  problem  can  be  overcome 
through  increased  fixturing  of  the  baskets  and  blades  in  addition  to  utilizing  software 
manipulation  of  the  image  data  to  align  the  images. 

4-3.4-1  Blade  Classification  Algorithm.  The  blade  classification  algorithm  is 
similar  to  the  one  used  for  alternative  2.  The  main  concern  is  to  collect  additional  features 
and  use  the  features  to  more  accurately  classify  the  blades.  The  system  will  utilize  the  set 
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of  features  collected  by  the  software  in  the  previous  alternative,  as  well  as  the  informa¬ 
tion  gained  from  viewing  the  blades  at  different  times.  The  expanded  feature  set  should 
be  sufficient  to  correctly  classify  approximately  99.5  percent  of  the  defective  blades  and 
maintain  a  false  call  rate  of  approximately  2.5  percent  (see  appendix  C). 

4.3.5  Expected  System  Performance.  The  lABR  system  should  be  capable  of  in¬ 
specting  at  least  1  blade  every  10  seconds  on  average  (3  seconds  for  data  collection  and 
processing  and  7  seconds  for  blade  manipulation).  The  system’s  false  call  rate  should  be 
no  greater  than  that  of  a  human  inspector  (2.5  percent)  with  a  probability  of  identifying  a 
crack  of  approximately  99.5  percent.  Table  4.3  summarizes  the  components  and  expected 
performance  of  alternative  3. 


Component 

Quantity 

AFPPM 

1 

lABR 

-  Manipulator  Sys 

2 

-  CCD  w/light  source 

2 

-  Master  Computer 

1 

Human  Inspectors 

0 

SQC 

-  Inspectors 

1 

-  Microscope  and  Blade  counter 

1 

Performance 

-  False  call  rate 

2.5  percent 

-  Avg  Inspection  Time 

10  sec/blade 

-  Probability  ID  Crack 

99.5  percent 

Table  4.3.  Summary  of  alternative  3. 


4.3.6  Models.  The  robotics  simulation  program,  IGRIP,  developed  by  Deneb  Robotics, 
Inc.  was  used  to  simulate  the  lABR.  The  simulation  used  for  this  alternative  is  the  same 
as  that  used  for  Alternative  2.  Appendix  F  provides  a  more  detailed  explanation  of  the 
robotic  simulation. 
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4-4  Summary. 

Each  of  the  alternatives  attempt  to  solve  the  problems  of  the  current  system  in  a 
different  manner.  Alternative  1,  manual  inspection,  attempts  to  solve  the  problems  by 
removing  the  source  of  the  majority  of  the  problems  (the  FPIM)  and  increasing  the  per¬ 
formance  of  the  remaining  system  through  an  SQC  plan.  This  alternative  will  potentially 
offer  significant  short  term  gains,  but  fails  to  completely  address  the  variablility  of  human 
inspector  performance. 

Alternative  2,  senai-automated  inspection,  attempts  to  solve  the  problems  of  the 
current  system  by  replacing  the  FPIM  with  an  improved  ABR.  The  new  ABR  will  be 
more  maintainable,  easier  to  use,  and  provide  more  accurate  results.  The  ABR,  along  with 
an  effective  SQC  plan,  also  aids  in  reducing  the  variability  of  the  system  by  decreasing 
the  workload  of  the  human  inspectors.  This  alternative  is  readily  realizable  and  resembles 
other  operational  FP  inspection  systems  (1,  51). 

Alternative  3,  completely  automated  inspection,  aims  to  further  reduce  the  variabil¬ 
ity  of  the  system  through  complete  automation.  A  totally  automated  blade  inspection 
system  allows  for  more  consistent  performance  by  providing  direct  control  over  the  system 
performance  parameters.  Alternative  3  potentially  offers  the  most  significant  long  term 
gains  due  to  its  accuracy  and  consistency  in  classifying  blades.  Table  4.4  summarizes  the 
alternatives. 

All  three  alternatives  developed  in  this  section  are  feasible  solutions  which  attempt 
to  satisfy  the  needs,  alterables,  and  constraints  identified  in  the  UPP.  Each  alternative  has 
its  associated  advantages  and  disadvantages.  In  order  to  evaluate  the  relative  merits  of  the 
alternatives,  the  value  system  developed  in  the  UPP  must  be  applied  to  each  alternative 
to  more  clearly  distinguish  them  from  each  other. 
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Components 

AFPPM 


Alt  1 


Alt  2 


Alt  3 


1 

2 

1 

2 

1 

1 

ABR 


-  Manipulator  Sys 


-  CCD  w/light  source 


-  Master  Computer 


Human  Inspectors 


SQC 


-  Inspectors 


-  Microscope  and  Blade  counter 


Performance 


-  False  call  rate  (percent)  2.5  I  5.5  I  2.5 


-  Avg  Inspection  Time  per  blade  6.25  sec  11  sec  10  sec 


-  Probability  ID  Crack  (percent)  90  99.5  99.5 


Table  4.4.  Summary  of  the  alternatives. 


V'.  System  Evaluation 


5. 1  Introduction 

The  evaluation  process  will  compare  the  baseline  system  and  each  alternative  against 
the  objectives  to  determine  their  relative  performance.  The  analysis  provides  an  estimate 
of  the  expected  performance  and  the  confidence  level  associated  with  each  design  and 
identifies  the  alternative  that  offers  the  best  potential  to  improve  the  current  inspection 
system. 

Recall  that  the  objective  of  the  systems  engineering  study  is  to  recommend  changes 
to  the  IBIS  that  will  improve  the  overall  FP  inspection  process  for  jet  engine  blades  at 
SA-ALC.  Chapter  II  introduced  the  problem  by  describing  the  present  inspection  system 
and  highlighting  areas  for  potential  improvement.  In  chapter  III,  the  systems  engineering 
methodology  was  applied  to  develop  a  framework  that  could  be  used  to  analyze  the  problem 
and  develop  alternatives.  The  framework  also  identified  five  key  objectives  of  the  study  and 
their  related  objective  measures  that  would  contribute  to  satisfying  the  overall  objective. 
With  these  objectives  in  mind,  the  design  alternatives  were  developed,  as  outlined  in 
chapter  IV.  Adequate  information  is  now  available  to  evaluate  the  design  alternatives  and 
recommend  the  best  course  of  action. 

The  decision  analysis  involves  several  steps  and  will  yield  a  variety  of  products  that 
capture  a  great  deal  of  information  about  the  design  alternatives  and  their  expected  per¬ 
formances.  The  evaluation  process  follows  the  outline  established  by  Athey  in  his  text. 
Systematic  Systems  Approach  (2).  The  process  determines  the  relative  importance  of  each 
objective  or  criterion  and  assigns  a  weight  to  it.  The  alternatives  are  then  evaluated  with 
respect  to  the  five  criteria  to  determine  how  well  each  criterion  is  satisfied.  For  each  cri¬ 
terion,  the  alternatives  are  rated  on  a  scale  of  0  to  10  as  a  measure  of  utility.  In  addition, 
the  analysis  characterizes  the  accuracy  of  the  information  used  to  evaluate  each  alternative 
by  assigning  a  confidence  factor  to  each  utility  rating.  The  confidence  factors  are  used  to 
discount  the  original  utility  value  in  the  final  evaluation. 

All  of  the  utility  values  and  confidence  factors  are  combined  into  an  evaluation  matrix 
which  summarizes  the  results  of  the  analysis  and  allows  the  best  alternative  to  be  selected. 
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The  alternative  with  the  highest  utility  is  usually  recommended,  but  the  presentation  of  all 
of  the  information  in  the  evaluation  matrix  allows  a  better  understanding  of  the  decision 
process. 

The  analysis  shows  that  design  alternative  3,  the  fully  automated  system,  has  the 
highest  overall  utility,  which  means  it  oflfers  the  best  potential  to  improve  system  perfor¬ 
mance.  However,  alternative  1,  with  entirely  manual  inspection,  has  the  highest  discounted 
utility  when  considering  the  confidence  factors  associated  with  each  design.  Even  though 
alternative  3  has  higher  utility,  its  low  confidence  rating  indicates  that  it  requires  approx¬ 
imately  five  years  of  effort  to  characterize  the  design  before  it  is  implemented.  The  best 
course  of  action  for  SA-ALC  to  improve  the  inspection  process  is  to  use  manual  inspection 
together  with  a  statistical  quality  control  plan  while  research  continues  to  improve  the  con¬ 
fidence  levels  in  the  semi-automated  and  fully  automated  design  concepts.  The  automated 
systems  offer  the  best  potential  for  future  inspection  systems  because  of  the  consistency 
of  the  process  and  the  low  number  of  system  errors  in  classifying  blades.  The  evaluation 
matrix  at  the  end  of  this  chapter  (table  5.25)  summarizes  all  of  the  relevant  information 
in  the  decision  process. 

5.2  Evaluation  Tools 

5.2.1  Preference  Chart.  Before  analyzing  the  alternatives  in  detail,  the  five  main 
criteria  are  compared  to  each  other  to  determine  their  relative  importance.  The  criteria 
were  introduced  as  objectives  in  chapter  III  and  are:  the  cost  associated  with  incorrectly 
classifying  blades,  the  system  maintainability,  the  system  cost,  the  system  throughput 
capacity,  and  the  potential  for  the  system  to  be  upgraded  in  the  future.  The  relative 
importance  is  calculated  using  a  preference  chart,  which  displays  pair-wise  comparisons 
between  all  of  the  criteria.  As  described  by  Athey,  the  chart  can  be  used  to  cissign  weights 
to  each  criterion  which  reflect  its  relative  value  in  the  decision  process  (2). 

Table  5.1  displays  the  symbols  used  to  relate  each  element  in  the  preference  chart 
and  table  5.2  shows  the  chart  developed  for  the  blade  inspection  criteria.  The  elements  on 
the  left  side  of  the  preference  chart  are  compared  to  the  elements  across  the  top  using  one 
of  the  five  symbols.  For  example,  the  maintainability  criterion  is  much  less  important  than 
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the  blade  misclassification  criterion,  so  the  C  symbol  was  entered  in  the  preference  chart 
on  the  maintainability  row  and  under  the  blade  misclassification  column.  The  chart  must 
be  consistent,  so  the  symbol  on  the  blade  misclassification  row  under  the  maintainability 
column  must  be  >,  which  is  the  inverse  of  the  previous  symbol.  Table  5.1  also  provides 
the  point  values  associated  with  each  comparison,  which  are  used  to  compute  the  weights 
for  the  criteria. 


Symbol 

Meaning 

Points 

> 

much  more 

4 

> 

more 

3 

= 

equal 

2 

< 

less 

1 

< 

much  less 
_ 1 

0 

Table  5.1.  Comparison  symbols  for  the  preference  chart. 


Table  5.2  shows  the  preference  chart  when  all  of  the  pair-wise  comparisons  are  com¬ 
pleted.  To  calculate  the  weight  associated  with  each  criterion,  the  point  values  for  each 
symbol  are  entered  into  the  chart  and  summed  across  each  row.  The  total  for  each  row  is 
the  raw  value  associated  with  the  corresponding  criterion.  These  raw  values  can  be  used 
directly  or  they  can  be  converted  into  any  convenient  format  for  the  analysis,  provided  the 
relative  magnitudes  of  the  values  remain  constant.  For  the  decision  analysis,  the  raw  values 
were  standardized  so  their  sum  equaled  100.  Note  that  the  relative  magnitudes  remained 
constant  between  the  raw  values  and  the  standardized  weights,  as  shown  in  table  5.2. 

The  most  important  criterion  for  the  analysis  is  the  cost  associated  with  incorrectly 
classifying  a  blade,  which  is  listed  in  the  chart  as  misclassifications  by  the  system.  The 
misclassification  category  received  a  weight  of  40.0,  which  is  almost  twice  as  large  as  the 
weight  for  the  second  most  important  criterion,  maintainability.  Following  maintainabil¬ 
ity  in  terms  of  importance  are  annual  cost,  throughput,  and,  finally,  upgradability.  The 
weights  play  a  significant  role  in  calculating  the  overall  utility  of  each  alternative  in  the 
final  evaluation  matrix. 
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Criteria 

Main 

Misclass. 

Thru 

Cost 

Upgrade 

Value 

Weight 

Misclassification 

> 

> 

» 

» 

16 

40.0 

Maintainability 

< 

> 

> 

> 

9 

22.5 

Annual  Cost 

< 

< 

> 

> 

7 

17.5 

Throughput 

< 

< 

< 

> 

5 

12.5 

Upgradability 

< 

< 

< 

< 

3 

BQI' 

Table  5.2.  Preference  chart. 


It  is  important  to  note  that  the  weights  developed  with  the  preference  chart  are 
completely  independent  of  the  of  the  utility  scores  and  confidence  factors  discussed  in  the 
following  section.  The  three  values  are  combined  within  the  evcJuation  matrix,  but  prior 
to  the  final  evaluation,  a  change  in  any  of  the  values  does  not  necessitate  a  change  in 
the  other  values.  This  fact  is  important  because  it  indicates  that  the  weights  can  easily 
be  altered  to  reflect  a  different  perspective  on  the  inspection  problem  without  affecting 
the  utility  scores  for  the  different  system  designs.  Appendix  L  illustrates  one  scenario  for 
adjusting  the  preference  chart  and  weights  and  characterizes  the  sensitivity  of  the  final 
evaluation.  In  addition,  section  5.9  summarizes  the  results  of  this  sensitivity  analysis. 

5.2.2  Utility  Measures.  With  the  preference  chart  and  relative  weights  established, 
each  alternative  now  must  be  evaluated  against  the  criteria  to  determine  how  well  it  satisfies 
the  objectives.  Following  Athey’s  approach,  each  criterion  is  developed  in  detail  and  the 
objective  measures  outlined  in  chapter  III  are  used  to  quantify  the  value  of  each  design 
alternative.  After  a  thorough  explanation  of  the  criterion,  a  utility  score  is  assigned  and 
justified  for  each  design  alternative.  The  utility  scores  range  from  0  to  10.  A  high  score 
indicates  the  alternative  satisfies  the  criterion  exceptionally  well  and  a  low  score  means 
the  alternative  is  lacking  in  that  particular  area.  .‘\s  shown  in  table  5.3,  Athey  developed 
the  following  labels  for  the  range  of  utility  values. 

Each  criterion  is  slightly  different  in  terms  of  how  it  measures  the  value  of  the  alterna¬ 
tives  or  how  it  quantifies  the  results.  For  this  reason,  the  criteria  are  presented  separately 
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Rating 

Utility  Score 

Exceptional 

9,  10 

Above  Average 

7,8 

Average 

4,  5,6 

2,  3 

Barely  Acceptable 

0,  1 

Table  5.3.  Utility  values  used  to  score  each  design  alternative  against  the  criteria. 

in  the  following  sections.  Although  the  analysis  was  partitioned  to  present  the  results  of 
the  evaluation,  the  criteria  are  actually  tightly  related  and  cannot  be  considered  indepen¬ 
dently.  For  example,  the  results  of  the  analysis  for  incorrectly  classifying  blades  determine 
the  number  of  good  and  defective  blades  sent  to  human  inspectors  for  each  alternative. 
The  number  of  blades  sent  to  the  inspectors  directly  influences  the  system  throughput,  as 
well  a.s  the  cost  to  operate  the  system.  Clearly,  the  interactions  between  the  criteria  must 
be  systematically  monitored  to  maintain  a  consistent  evaluation. 

5.2.3  Assumptions  and  Confidence  Levels.  As  another  step  to  maintain  consistency 
in  the  evaluation  process,  a  variety  of  assumptions  were  made  about  the  system  configu¬ 
rations  and  their  actual  performance.  All  of  the  criteria  rely  on  the  assumptions  to  help 
complete  the  analysis.  Making  assumptions  allows  for  quantitative  calculations  in  area^ 
where  the  actual  information  was  not  accessible  during  the  course  of  this  study.  These 
assumptions  are  based  on  the  best  available  information  from  SA-ALC,  research  in  the 
appropriate  areas,  and  the  design  models  developed  during  the  study.  With  the  following 
assumptions  held  constant  for  each  criterion,  a  more  equitable  comparison  can  be  made 
between  the  alternative  designs. 

Assumptions; 

1.  Each  system  will  inspect  1 .2  million  blades  per  year.  This  figure  is  substantially  larger 
than  the  present  number  of  blades  inspected,  but  it  is  in  line  with  the  anticipated 
number  of  blades  in  future  years  according  to  SA-ALC. 

2.  Five  percent  of  all  blades  are  actually  defective.  The  management  at  SA-ALC  made 
this  estimate  based  on  the  percentage  of  parts  identified  as  defective. 
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3.  A  human  inspector  requires  5  seconds  to  inspect  a  good  blade  and  30  seconds  to 
inspect  a  bad  blade.  These  times  are  estimates  based  on  observing  an  inspector  in 
action  and  feedback  from  SA-ALC. 

4.  For  the  baseline  system,  the  FPIM  receives  approximately  5%  of  the  parts  pro¬ 
cessed  through  the  AFPPM.  SA-ALC  provided  data  for  calendar  year  1991  showing 
that  3.41%  of  the  parts  processed  through  the  AFPPM  were  sent  to  the  FPIM. 
The  value  5%  was  selected  for  the  calculations,  because,  according  to  SA-ALC,  the 
percentage  of  blades  sent  to  the  FPIM  in  1991  was  lower  than  average. 

5.  For  the  baseline  system,  the  FPIM  has  a  50%  probability  of  labeling  a  good  blade 
as  defective  (false  call)  and  a  1%  probability  of  labeling  a  cracked  blade  as  good 
(missed  detection).  The  1991  data  showed  that  the  FPIM  labeled  58.48%  of  all 
blades  as  suspect.  Personnel  at  SA-ALC  estimated  that  the  FPIM  detects  99%  of 
all  bad  blades.  Therefore,  50%  and  1%  represent  reasonable  estimates  for  the  false 
call  and  missed  detection  error  rates. 

6.  For  alternative  2,  the  automated  system  labels  20%  of  the  good  blades  as  sus¬ 
pect.  The  preliminary  work  with  the  image  processing  and  decision  analysis  software 
showed  the  percentage  of  suspect  blades  would  be  in  the  range  of  15  to  30%.  20% 
was  selected  as  a  reasonable  estimate. 

7.  For  both  alternatives  2  and  3,  the  automated  system  requires  10  seconds  to  inspect 
one  blade.  The  robotic  simulation  estimated  that  approximately  7  seconds  would  be 
reqmred  to  manipulate  the  blade  while  presenting  it  to  the  sensor.  The  additional  3 
seconds  are  for  the  image  processing  and  decision  analysis.  For  alternative  3,  the  10 
second  time  represents  the  rate  at  which  blades  leave  the  second  inspection  station. 
This  is  a  conservative  estimate. 

8.  Finally,  the  calculations  assume  all  blades  are  properly  processed  for  inspection.  In 
other  words,  a  human  inspector  can  successfully  process  each  blade  without  repro¬ 
cessing  it  through  the  AFPPM. 

Even  with  the  above  assumptions,  part  of  the  evaluation  analysis  is  qualitative  in 
nature  because  the  scope  of  the  study  limited  the  amount  of  research  that  could  be  per¬ 
formed.  Qualitative  measures  were  unavoidable  for  this  type  of  design  problem.  It  may 
seem  artificial  to  quantify  criteria  that  are  inherently  subjective,  but  part  of  the  evalu¬ 
ation  process  compensates  for  this  problem.  During  the  evaluation  of  each  alternative, 
a  confidence  factor  is  developed  for  each  utility  score.  The  confidence  factor  represents 
the  accuracy  of  the  information  or  the  amount  of  faith  in  a  particular  utility  value.  The 
confidence  levels  range  from  0  to  1.  Athey  provides  a  description  for  four  levels,  as  shown 
in  table  5.4. 

The  evaluation  of  each  alternative  uses  the  values  in  table  5.4  as  a  guide  to  assign 
a  confidence  factor  to  each  utility  value.  At  the  completion  of  the  evaluation  process,  the 
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Table  5.4.  Confidence  levels  used  in  evaluating  alternatives. 


confidence  factors  are  used  as  multipliers  to  discount  the  utility  values  for  each  alternative. 
Used  in  this  way,  the  confidence  factors  reduce  the  effective  utility  of  the  aspects  of  the 
design  that  have  high  uncertainty.  In  the  final  evaluation,  each  alternative  receives  a 
raw  utility  score,  a  discounted  utility  score,  and  an  overall  confidence  level.  The  overall 
confidence  level  can  be  used  to  indicate  the  amount  of  research  that  must  be  done  to  verify 
the  concepts  in  the  corresponding  alternative. 

The  preference  chart  along  with  the  information  about  utility  values,  design  variables, 
assumptions,  and  confidence  levels  provide  an  introduction  to  the  evaluation  process.  The 
analysis  will  consider  each  criterion  separately  to  quantify  the  value  of  the  alternatives  and 
the  certainty  of  the  associated  information.  After  developing  all  of  the  criteria,  the  results 
are  summarized  in  the  final  evaluation  matrix  and  a  single  alternative  is  identified  as  the 
best  solution  to  the  jet  engine  blade  inspection  problem. 


5.3  Cost  of  Incorrectly  Classifying  Blades 


Not  detecting  a  cracked  blade  during  inspection  may  cause  jet  engine  damage  which 
has  an  associated  cost  to  the  USAF.  Appendix  D  includes  an  analysis  of  the  cost  of  incor¬ 
rectly  classifying  blades.  The  results  of  this  analysis  indicate  that  it  is  considerably  more 
probable  that  a  cracked  blade  will  result  in  an  unscheduled  maintenance  event  than  in  a 
weapon  system  loss.  Using  the  cost  model  developed  in  appendix  D,  the  cost  of  incorrectly 
classifying  blades  (CICB)  values  were  calculated  for  the  baseline  and  each  alternative. 
The  CICB  values  represent  the  predicted  annual  cost  to  the  AF  due  to  future  unscheduled 
maintenance  events  caused  by  the  missed  detection  of  cracked  blades  and  the  additional 
cost  of  discarding  good  blades. 
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It  is  possible  to  incorrectly  classify  blades  in  two  manners,  each  with  an  associated 
cost.  First,  it  is  possible  to  call  a  bad  blade  good  (h-ff ).  Making  this  type  of  error  may  lead 
to  an  unscheduled  maintenance  event  as  mentioned  in  appendix  D.  From  this  appendix, 
the  cost  of  an  unscheduled  maintenance  event  is  calculated  to  be  $  58.5K,  which  includes  all 
base  and  depot  costs.  The  second  type  of  error  is  calling  good  blades  bad  (g-b).  Assuming 
no  scrap  value,  each  good  blade  called  bad  will  cost  approximately  $  300. 

The  results  from  the  CICB  model  are  shown  in  table  5.5.  In  addition  to  the  number 
of  incorrect  classifications  for  each  type  of  error  mentioned  above,  this  table  also  displays 
the  number  of  good  blades  called  good  {g-g)  and  the  expected  number  of  bad  engines 
assembled.  The  expected  number  of  bad  engines  assembled  is  considerably  less  than  the 
number  of  bad  blades  called  good  because  of  the  probability  of  putting  more  than  one  bad 
blade  in  any  one  engine. 


Name 

9-g 

Total  bad  engines 

g-b 

Cost 

Baseline 

1.112  M 

6027 

1480 

27790 

$94.90  M 

Alt  1 

1.112  M 

6000 

1478 

28500 

$95.03  M 

Alt  2 

1.077  M 

330 

295 

62700 

$36.08  M 

Alt  3 

1.112  M 

300 

295 

28500 

$24.46  M 

Table  5.5. 


Initial  calculations  for  the  cost  of  incorrectly  classifying  blades  calculations 
assuming  1.2  M  blades  inspected  and  the  probability  that  a  blade  is  bad  of 


.05 


Several  assumptions  were  needed  to  calculate  the  CICB  for  each  system  and  are 
shown  in  tables  5.6  and  5.7. 


Name 

Baseline 

Alt  1 

Alt  2 

Alt  3 

PIC  Human 

.9 

.9 

.95 

NA 

PIC  Machine 

.99 

NA 

.99 

.995 

False-Calls  Human 

.025 

.025 

.025 

NA 

False-Calls  Machine 

.50 

NA 

.25 

.025 

Table  5.6.  False  call  and  PIC  assumptions  for  the  CICB  calculations 
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Assumption 

Value 

N  umber  of  Blades 

1.2  M 

Probability  that  a  blade  is  bad 

.05 

Cost  of  an  unscheduled  maintenance  event 

$58.5K 

Blade  replacement  cost 

$300 

Table  5.7.  System  assumptions  for  the  CICB  calculations 


Appendix  D  provides  detailed  information  on  why  these  assumptions  are  reasonable 
and  their  effect  on  the  CICB  calculations.  In  order  to  maintain  consistency  with  other 
criteria,  the  CICB  values  for  each  system  must  be  assigned  weights  and  confidence  factors. 
Sections  5.3.1  and  5.3.2  describes  the  assignment  of  confidence  values  and  weights  to  each 
alternative  based  on  the  CICB  calculations.  Following  these  discussions,  section  5.3.3 
provides  a  brief  overview  of  the  sensitivity  analysis  done  in  appendix  D. 

5.S.1  Confidence  Factors.  There  is  a  certain  amount  of  confidence  in  the  operating 
characteristic  data  derived  or  obtained  for  each  inspection  system  examined.  The  confi¬ 
dence  factor  is  not  only  based  on  the  reliability  of  the  source  of  information,  but  also  on 
the  belief  that  the  data  is  not  subject  to  change  and,  in  the  cases  were  the  data  is  ba.sed  on 
futuristic  conditions,  that  the  data  is  realizable.  For  each  system  the  following  paragraphs 
provide  justification  for  the  confidence  values  assigned  in  table  5.12. 

For  the  baseline,  the  values  used  for  PIC  and  FC  are  verifiable  from  historical  data. 
Because  of  this,  confidence  in  these  values  is  fairly  high.  However,  there  is  neither  a 
guarantee  that  the  system  is  still  operating  at  these  values,  nor  is  there  a  procedure  in- 
place  to  insure  the  system  operates  within  design  parameters.  Both  of  these  considerations 
contribute  to  an  assignment  of  an  overall  confidence  value  of  .6. 

Remember  that  alternative  1  only  uses  humans  for  the  inspection  task.  As  with  the 
baseline,  there  is  historical  data  to  substantiate  the  PIC  of  the  human,  and  despite  not 
knowing  if  this  data  is  still  valid,  the  suggested  SQC  plan  should  increase  the  confidence  of 
the  actual  inspection  values  and  provide  a  metric  to  insure  the  system  does  not  fall  outside 
of  established  operating  parameters.  Appendix  B  details  an  SQC  plan  appropriate  for  the 
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IBIS  process.  Based  on  these  arguments,  alternative  number  1  received  a  confidence  value 
of  .9. 

Assignment  of  confidence  values  for  alternative  2  is  complicated  by  the  use  of  both 
human  and  automated  inspection.  From  the  above  argument  for  alternative  1,  the  confi¬ 
dence  in  the  human  inspector  data  is  fairly  high.  In  addition,  from  appendix  I,  the  manual 
inspector’s  PIC  is  expected  to  increase  due  to  decreased  workload.  For  the  automated 
inspection  system,  the  PIC  and  FC  values  used  are  based  on  the  results  of  the  neural  net¬ 
work  evaluation  discussed  in  appendix  G.  Confidence  for  this  alternative  is  increased  due 
to  the  promising  results  presented  in  appendix  C  which  shows  a  feasible  manner  to  extract 
additional  features  from  cracked  blades.  Preliminary  research  shows  a  neural  network  or  a 
means  of  detecting  a  unique  characteristic  of  a  cracked  blade  can  be  exploited  to  establish 
the  values  shown  in  table  5.6.  Also,  the  use  of  an  SQC  plan  for  this  alternative  will  identify 
system  performance  degradation.  Because  of  these  considerations,  a  confidence  factor  of  .5 
is  assigned  to  this  alternative. 

The  confidence  value  for  alternative  3  is  .5.  This  confidence  value  is  reasonable  be¬ 
cause  of  the  development  needed  for  this  alternative.  With  additional  research  as  described 
in  appendix  C,  the  PIC  and  FC  rate  for  this  alternative  may  be  realized.  However,  the 
neural  network  analysis,  see  appendix  G,  presents  a  tradeoflF  between  false  calls  and  missed 
detections.  From  the  presentation  in  this  appendix,  the  operating  point  desired  for  alter¬ 
native  3  in  terms  of  inspection  error  rates  is  not  yet  available.  This  is  not  to  say  that 
neural  networks  are  incapable  of  performing  well  enough  to  attain  alternative  3,  but  the 
results  from  the  data  presented  indicate  that  more  research  must  be  performed.  As  with 
the  other  alternatives,  the  recommendation  to  use  SQC  to  measure  system  operating  point 
increases  the  confidence  in  this  alternative. 

5.3.2  CICB  Weights.  To  consider  the  CICB  values  with  other  criteria,  it  is  nec¬ 
essary  to  use  the  same  metric.  Converting  the  dollar  values  from  table  5.5  into  weights 
requires  the  use  of  hypothetical  best  and  worst  case  systems.  Using  the  model  from  ap¬ 
pendix  D  and  the  assumptions  in  table  5.7,  values  for  the  hypothetically  best  and  worst 
case  systems  defined  in  table  5.8  were  obtained.  Linearizing  the  inspection  system  between 
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these  weights,  assuming  the  best  gets  a  score  of  10  and  the  worst  a  score  of  0,  yields  the 
results  shown  in  table  5.9.  The  utility  score  for  each  alternative  was  calculated  by  assuming 


System 

PIC 

FC 

Cost 

Worst 

$  114.9  M 

Best 

$  6.862  M 

Table  5.8.  CICB  calculations  for  the  hypothetically  best  and  worst  case  systems 


a  linear  relationship  between  expected  cost  and  utility.  The  best  system  was  assumed  to 
have  a  utility  score  of  10  and  the  worst  system  received  a  score  of  0.  The  utility  values  for 
the  baseline  system  and  the  alternatives  were  then  scaled  appropriately  between  the  best 
and  worst  values. 


System 

Weight 

Baseline 

1.85 

Alt  1 

1.839 

Alt  2 

7.29 

Alt  3 

8.37 

Table  5.9.  CICB  weight  assignments.  Best  possible  score  is  a  10,  worst  is  0. 


5.3.3  Sensitivity  Analysis.  As  mentioned  in  appendix  D,  the  CICB  calculations 
change  if  the  assumptions  used  in  the  CICB  model  are  varied.  Two  assumptions  that  are 
main  drivers  in  the  cost  calculation  are  the  probability  that  a  blade  is  bad  and  the  number 
of  blades  inspected.  Originally,  the  probability  that  a  blades  is  bad  was  assumed  to  be  .05 
and  the  number  of  blades  needing  inspection  was  1.2  M.  Assuming  the  probability  that 
a  blade  is  bad  is  .007  and  the  number  of  blades  inspected  is  392,518,  the  CICB  model 
was  reevaluated.  These  two  values  were  chosen  based  on  the  number  of  compressor  blades 
actually  inspected  during  one  year  and  the  number  of  blades  identified  by  the  IBIS  as 
bad  in  comparison  to  the  total  number  inspected.  Table  5.10  displays  the  results  of  this 
reevaluation.  In  this  table,  the  baseline  validation  row  lists  results  from  the  validation 
effort  in  appendix  D.  Since  the  validation  effort  did  not  include  the  costs  associated  with 
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calling  good  blades  bad,  table  5.10  contains  only  the  expected  costs  due  to  unscheduled 
maintenance  events. 


Name 

Total  bad  engines 

Cost  due  to  bad  engines 

Baseline  CICB 

213 

$12.46  M 

Alt  1 

213 

$12.46  M 

Alt  2 

15 

$.877  M 

Alt  3 

14 

$.819  M 

Baseline  Validation 

62 

$3.63  M 

Table  5.10.  Cost  of  assembling  bad  engines  using  the  probability  of  a  blade  being  bad  as 
.007  and  total  number  of  blades  to  inspect  equal  to  392,518. 


An  important  feature  of  the  data  listed  in  table  5.10  is  that  when  the  weights  are 
linearized  in  a  similar  manner  as  was  done  in  table  5.9,  the  relative  weightings  of  the 
alternatives  does  not  change  considerably  as  shown  in  table  5.11.  This  example  provides 
evidence  that  the  evaluation  of  the  alternatives  is  not  extremely  sensitive  to  assumptions 
made  in  the  CICB  analysis. 


System 

Weight 

Baseline 

.455 

Alt  1 

.455 

Alt  2 

9.45 

Alt  3 

9.5 

Table  5.11.  CICB  weight  assignments  considering  only  unscheduled  maintenance  events 
and  a  probability  that  a  blade  is  bad  of  .007.  Best  possible  score  is  a  10, 
worst  is  0. 

5.3.4  Summary.  The  expected  cost  of  incorrectly  classifying  blades  for  each  alter¬ 
native  was  analyzed  and  used  as  an  evaluation  criterion.  A  brief  overview  of  appendix  D, 
which  contains  more  information  on  the  CICB  was  also  presented.  Since  several  of  the 
results  from  this  appendix  depended  on  assumptions,  the  assignment  of  confidence  factors, 
which  will  be  used  in  the  decision  analysis,  was  performed.  In  addition,  the  cost  values 
for  the  CICB  were  linearized  and  given  weights  to  facilitate  comparison  with  other  crite- 
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ria.  Finally,  sensitivity  analysis  was  presented  demonstrating  the  robustness  of  the  relative 
ranking  of  the  alternatives  for  the  CICB  calculations. 


Criterion 

Alternatives 

Expected  Outcome 

Confidence 

Support 

Cost 

Value 

Level 

Rating 

Baseline 

$  94.9  M 

5 

C 

0.6 

SA-ALC 

data  about  human  inspec¬ 
tor  performance. 

Martin  Marietta’s  study 

POD  for  humans  and 

FPIM 

Cost 

of  Incor¬ 
rectly 
Classify¬ 
ing 

Blades 

1.  Manual 
Inspection 

$  95.03  M 

5 

VC 

0.9 

Based  on  same  informa¬ 
tion  used  to  estimate  the 
baseline. 

Assumes  human  perfor¬ 
mance  remains  constant 

without  the  FPIM. 

2.  Semi- 
Automated 

$  36.08  M 

1 

C 

0.5 

Expected  increase  in  hu¬ 
man  performance  consid¬ 
ering  human  factors. 

Image  processing  and  clas¬ 
sification  software  model 

to  estimate  error  rate  per¬ 
formance. 

3.  Fully 
Automated 

$  24.46  M 

9 

C 

0.5 

Expected  improvements  in 
neural  network. 

Exploitation  of  additiontJ 

information  in  FP  indica¬ 
tion. 

Table  5.12.  Summary  of  the  cost  of  incorrectly  classifying  blades. 


5.4  Maintainability  Assessment 

5.4.1  Introduction.  System  maintainability  is  the  second  most  important  criterion 
used  to  evaluate  the  design  alternatives.  There  are  two  primary  reasons  for  selecting  system 
maintainability  as  a  criterion.  First,  system  maintainability  contributes  substantially  to 
the  overall  operating  costs.  As  a  contributing  factor  to  the  cost,  system  maintainability 
represents  an  element  of  the  cost  that  can  be  controlled  in  the  system  design.  Second,  the 
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inspection  system  must  have  sufficient  availability  to  assure  mission  success.  Availability 
is  adequate  if  preventive  maintenance  succeeds  in  controlling  failures  and  if  corrective 
maintenance  expeditiously  repairs  the  system  when  failures  do  occur.  If  the  system  is 
available,  the  system  throughput  requirements  can  be  achieved.  Mission  success  results 
from  inspecting  and  correctly  classifying  blades. 

The  system  maintainability  assessment  criterion  consists  of  three  maintainability 
measures:  maintenance  burden,  system  modularity,  and  diagnostic  capability.  The  main¬ 
tenance  burden  is  defined  as  the  number  of  maintenance  personnel  the  system  requires  and 
their  respective  training  level.  This  measure  reflects  the  ease  of  fulfilling  the  maintenance 
plan  for  each  alternative. 

The  second  maintainability  measure  is  system  modularity.  A  system  depends  on 
functional  units  which  accept  inputs  from  and  provide  outputs  to  other  functional  units.  If 
the  system  hardware  reflects  this  logical  division  in  its  subsystems,  the  system  is  modular. 
System  modularity  increases  as  the  logical  division  of  hardware  subsystems  proceeds  to 
lower  levels  of  the  hardware  hierarchy.  The  system  modularity  reflects  the  ease  of  diagnosis 
and  repair  of  the  alternative.  Increased  modularity  emerges  as  a  maintainability  design 
goal  (see  Appendix  E). 


The  third  maintainability  measure  is  system  diagnosis  capability.  This  measure  ac¬ 
counts  for  the  completeness  and  the  ease  of  implementing  system’s  self  diagnosis  capability. 
An  increased  built  in  test  capability  also  emerges  as  a  maintainability  design  goal  (see  Ap¬ 
pendix  E). 


5.4.2  Maintainability  Value  Derivation.  The  baseline  system  and  each  of  the  three 
alternatives  will  receive  a  system  maintainability  utility  value.  The  most  maintainable 
alternative  will  receive  the  highest  score.  The  final  maintainability  value  is  the  weighted 
sum  of  the  numerical  scores  assigned  to  the  three  maintainability  measures.  The  prefer¬ 
ence  chart  method  was  used  to  develop  the  weighting  coefficients  to  combine  the  three 
maintainability  measures.  The  weighting  coefficients  are  .67  for  maintenance  burden.  .25 
for  system  modularity,  and  .08  for  system  diagnostic  capability,  as  shown  in  table  5.13. 


Note  that  the  weights  were  normalized  so  that  they  sum  to  one. 
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Table  5.13.  Preference  chart  for  the  maintainability  measures. 


The  maintenance  burden  represents  the  most  significant  measure  because,  if  the 
maintenance  burden  is  high,  preventive  and  corrective  maintenance  actions  will  be  diffi¬ 
cult  and  expensive.  Modularity  emerged  as  more  significant  than  self  diagnosis  capability 
because  a  modular  system  can  compensate  for  incomplete  diagnosis  capability.  The  weights 
in  table  5.13  will  be  used  to  combine  the  three  measures  to  compute  a  single  utility  value 
for  maintainability. 


5.4-S  Maintainability  Scores.  The  maintainability  scores  for  the  baseline  system 
were  assigned  first  and  then  used  to  generate  the  scores  for  the  three  design  alternatives. 
The  potential  improvements  offered  by  the  alternatives  were  compared  to  the  baseline 
system  and  an  appropriate  maintainability  score  was  developed. 

The  bciseline  system  received  a  score  of  3  for  the  maintenance  burden  category  be¬ 
cause  it  requires  an  experienced  NDI  engineer,  a  computer  engineer,  and  two  experienced 
technicians.  The  baseline  system  requires  individuals  specifically  trained  to  operate  the 
FPIM  because  of  the  unique  nature  of  the  system.  Alternative  1  received  a  maintenance 
burden  score  of  7  because  it  should  require  one  experienced  NDI  engineer  and  two  expe¬ 
rienced  technicians.  Alternative  2  received  a  maintenance  burden  score  of  5  because  it 
should  require  one  experienced  NDI  engineer,  one  experienced  computer  engineer  and  one 
experienced  technician.  Alternative  3  received  a  maintenance  burden  score  of  4  because 
it  should  require  one  experienced  NDI  engineer,  one  experienced  computer  engineer,  and 
two  experienced  technicians.  Although  the  personnel  requirements  for  the  baseline  system 
and  alternative  3  are  similar,  alternative  3  received  a  slightly  higher  score  because  the 
fully  automated  inspection  system  follows  industry  standards  and  there  should  be  fewer 
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demands  on  the  NDI  engineer.  Table  5.14  summarizes  the  personnel  requirements  for  each 
alternative  and  the  maintenance  burden  scores. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

NDI  Engineer 

1 

1 

1 

1 

Technicians 

2 

2 

1 

2 

Computer  Engineer 

1 

0 

1 

1 

Main.  Burden  Score 

_ 3 _ 

_J_J 

5 

4 

Table  5.14.  Personnel  required  to  operate  each  design  alternative.  The  final  row  shows 
the  maintenance  burden  scores. 


The  baseline  system  received  a  score  of  2  for  the  modularity  category.  The  FPIM 
substantially  contributes  to  the  low  score  because  it  is  not  modular.  In  addition,  the  AF- 
PPM  is  modular  at  the  subsystem  level,  but  its  modularity  ends  there.  The  components 
of  each  of  the  seven  AFPPM  stages  are  not  modular.  Alternative  1  received  a  modularity 
score  of  4  because  it  is  primarily  composed  of  the  AFPPM  and  human  inspectors.  Al¬ 
ternatives  2  and  3  both  received  a  modularity  score  of  4  because  they  are  composed  of  a 
modular  computer,  and  modular  manipulation  and  sensor  subsystems  that  must  work  in 
conjunction  with  the  AFPPM.  The  AFPPM  restricts  the  modularity  score  for  Alternatives 
2  and  3. 

The  bjiseline  system  and  alternative  1  received  a  score  of  3  for  the  self  diagnostic 
category  because  the  AFPPM  has  limited  self  diagnostic  capabilities.  Alternatives  2  and  3 
received  a  self  diagnostic  score  of  6  because  the  computer  system  possesses  the  capability  to 
assess  the  functionality  of  the  entire  IBIS.  Table  5.15  summarizes  all  scores  and  confidence 
factors  for  the  maintainability  analysis. 

5.4-4  Confidence  Factors.  Analyzing  the  present  inspection  configuration  provided 
confidence  scores  for  the  baseline  system.  Two  engineers  and  two  technicians  comprise  the 
current  maintenance  team,  so  the  maintenance  burden  is  known  exactly  and  the  associated 
confidence  factor  is  0.9.  The  confidence  factors  for  both  the  modularity  and  self  diagnostic 
measures  are  0.6,  because  information  on  the  current  system  configuration  did  not  clearly 
specify  how  the  system  performs  in  these  areas. 


5-16 


Category 

Weights 

Baseline 

Alt  1 

Alt  2 

Alt  3 

Score 

Conf. 

Score 

Conf. 

Score 

Conf. 

Score 

Conf. 

Maintenance  Burden 

.67 

3 

(.9) 

6 

(.6) 

5 

(.6) 

4 

(.3) 

Modularity 

.25 

2 

(.6) 

4 

(.6) 

4 

(.6) 

4 

(.6) 

Self  Diagnostics 

.08 

3 

(.6) 

3 

(.6) 

6 

(.6) 

6 

(.6) 

Maintainability  Value 

2.75 

(.8) 

5.25 

(.6) 

4.83 

(.6) 

4.17 

(•4) 

Table  5.15.  Maintainability  analysis.  The  confidence  levels  for  each  score  are  shown  in 
parentheses. 


The  baseline  and  alternative  1  have  similar  configurations.  The  primary  difference 
between  the  designs  is  that  alternative  1  removes  the  FPIM  from  the  system.  Without 
the  FPIM,  the  personnel  requirements  would  probably  decrease  to  one  engineer  and  two 
technicians.  Removing  the  FPIM  should  also  improve  the  system  modularity,  but  there 
probably  will  not  be  a  significant  change  in  the  self  diagnostic  capability,  as  shown  by  the 
scores  in  table  5.15.  The  confidence  factors  for  alternative  1  were  all  0.6,  because  there  is 
a  fair  level  of  certainty  about  the  estimated  changes. 

Alternative  2  developed  the  semi-automated  inspection  system  for  the  IBIS.  The 
analysis  determined  that  the  maintenance  burden  would  be  two  engineers  and  one  tech¬ 
nician.  One  engineer  would  have  primary  responsibility  over  the  ABR  maintenance  and 
secondary  responsibility  regarding  AFPPM  maintenance.  The  other  engineer  and  the  tech¬ 
nician  would  be  responsible  for  the  AFPPM  maintenance.  The  design  requires  that  the 
ABR  be  modular  and  have  self  diagnostic  capabilities.  As  with  alternative  1,  there  is  a 
fair  amount  of  confidence  in  the  alternative  2  design  and  its  impact  on  maintainability,  so 
each  of  the  three  mecisures  received  a  confidence  score  of  0.6. 

The  third  design  alternative  relies  on  a  fully  automated  inspection  system  to  process 
the  blades.  This  alternative  will  require  one  engineer  and  one  technician  to  maintain  the 
system  and  the  related  components.  Alternative  3  also  requires  one  engineer  and  one 
technician  to  maintain  the  AFPPM.  There  is  low  confidence  in  the  maintenance  burden 
score  for  alternative  3  because  the  inspection  system  has  not  been  completely  defined 
and  may  possess  a  different  maintenance  burden.  Accordingly,  the  maintenance  burden 
measure  received  a  confidence  factor  of  0.3  for  alternative  3.  The  analysis  for  modularity 


5-17 


Criterion 


Alternatives 


Expected  Outcome  Confidence 
Value  Level  Rating 


Support 


Maintain¬ 

ability 

Baseline 

2.8 

C  0.8 

l~ 

The  baseline  system  con¬ 
figuration  is  known. 

1.  Manual 
Inspection 

5.3 

C  0.6 

Alt  1  has  the  same  con¬ 
figuration  as  the  baseline 
without  the  FPIM. 

The  analysis  estimates  the 

maintentmce  burden  asso¬ 
ciated  with  the  FPIM  cis 

a  proportion  of  the  burden 
for  the  entire  system. 

2.  Semi- 
Automated 

4.8 

C  0.6 

Alt  2  introduces  the  ABR 
whose  design  will  be  mod¬ 
ular  and  will  have  self  di¬ 
agnostic  capabilities. 

It  is  assumed  that  one 

engineer  will  address  the 

ABR  maintenance  needs. 

3.  Fully 
Automated 

4.2 

LC  0.4 

Alt  3  introduces  the 
lABR. 

It  is  assumed  that  one 

engineer  and  one  techni- 

cieui  will  E  'dress  the  lABR 

maintenmice  needs. 

Table  5.16.  Summary  of  maintainability  evaluation. 


5-18 


and  self  diagnostic  capability  was  the  same  as  that  for  alteiuative  2,  so  these  measures 
received  a  confidence  factor  of  0.6  in  alternative  3  as  well. 

5.4.5  Maintainability  Summary.  The  final  results  of  the  maintainability  analysis 
should  not  be  surprising.  The  baseline  system  received  the  lowest  overall  maintainability 
score  primarily  because  of  the  presence  of  the  FPIM.  The  FPIM  represents  the  major 
difference  between  the  baseline  and  alternative  1.  In  addition,  alternative  1  contains  the 
fewest  mechanical  components  of  the  four  systems  considered,  so  it  received  the  highest 
maintainability  score.  Alternative  2  has  a  semi-automated  inspection  system,  while  alter¬ 
native  3  has  a  fully  automated  system  with  advanced  processing  and  additional  sensors. 
As  a  system  grows  in  complexity  it  becomes  more  difficult  to  maintain,  which  was  also 
reflected  in  the  results  of  the  maintainability  analysis.  Overall,  none  of  the  designs  received 
an  exceptionally  high  score  because  they  all  rely  on  the  AFPPM  to  prepare  blades  for  in¬ 
spection.  Replacing  the  AFPPM  with  a  more  maintainable  system,  should  be  considered 
to  improve  the  overall  performance  of  inspection  system. 

The  information  generated  by  the  maintainability  analysis  impacts  the  cost  analysis 
presented  in  the  following  section.  The  maintenance  analysis  characterizes  the  difficulty  in 
maintaining  the  system  in  question.  From  this  information,  the  cost  analysis  must  estimate 
the  annual  cost  to  maintain  the  inspection  systems. 

5.5  Annual  Cost 

To  estimate  the  annual  cost  of  each  alternative,  an  equivalent  uniform  annual  cost 
(EUAC)  was  calculated  for  each  system  based  on  an  interest  rate  of  10%  and  an  expected 
life  of  10  years.  The  EUAC  was  calculated  by  adding  the  operating  costs  for  one  year  to 
the  effective  annual  payment  that  would  be  required  to  amortize  the  initial  costs  over  the 
life  of  the  system  with  the  given  interest  rate.  Since  the  bulk  of  the  costs  associated  with 
each  system  are  annual  costs,  the  interest  rate  and  lifetime  are  not  critical  in  the  decision 
process.  The  costs  were  divided  into  operating  costs  and  capital  costs  and  the  Quatro  Pro 
spreadsheet  software  wa^  used  to  calculate  the  uniform  annual  net  costs.  A  printout  of  the 
spreadsheet  output  is  shown  in  table  5.17. 
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Operating 

Baseline 

Alt  1 

Alt  2 

Expendabl 

Seals 

500 

Penetrant 

21600 

Eaulaifie 

9600 

Developer 

3600 

limadiea 

2000 

llgicide 

65 

Silicon 

40 

Bnlbs 

450 

Solenoids 

260 

Printer 

760 

Salt 

1300 

Freon 

21600 

Verilon 

200 

Disks 

100 

Snakes 

180 

Tapes 

750 

Film 

552 

Total 

63557 

63557 

63557 

Spares 

Blackligh 

1800 

BloeerFen 

45 

Clutch 

110 

Timer 

12 

Soap  Feed 

340 

Starters 

1600 

Vibrator; 

250 

Hard  Disk 

320 

Monitors 

1500 

Heaterpar 

250 

Amp  Board 

4500 

Laser 

7500 

Disk  Driv 

350 

Total 

18577 

10000 

12000 

Total 

197121.6  176536.8  181336.8 

Capital 

Develop 

Softsare 

0 

0 

859164 

Hardware 

0 

6400 

213000 

Total 

0 

6400 

1072164 

Sjstens 

Computer 

0 

0 

100000 

Robot 

0 

0 

61000 

Sensor 

0 

0 

20000 

sqc 

0 

32000 

32000 

Total 

0 

32000 

213000 

Total 

0 

38400 

1285164 

Peraonnel 

Contract 

358905.8  168685.7 

204576.3 

Technicians 

681576  683487.8  598808.2 

SQC  Personnel 

0 

157248 

157248 

Total 

1040482 

1009422  960632.5 

Totals 

Annual 

1237603 

1185958 

1141969 

Init ial 

0 

38400 

1285164 

LCC  Costs 

7603835 

7324928 

8301423 

me 

1237603 

1192208 

1351124 

ScaledEUlC 

7 

.503617  8.105422 

5.998689 

bast 

1049295  vorst 

1803620 

Table  5.17.  Cost  Calculations 


lit  3 


635S7 


15000 

188536.8 


1036469 

319000 

1355469 


125000 

122000 

40000 

32000 

319000 

1674469 


240466.9 
583488 
157248 

981202.9 


1169740 

1674469 

8861350 

1442252 

4.790615 
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Some  of  the  costs  in  the  table  require  a  brief  explanation.  The  costs  for  the  expend¬ 
able  materials  and  spare  parts  for  the  baseline  system  were  provided  by  SA-ALC.  The 
quantity  of  materials  required  is  directly  proportional  to  the  number  of  parts  processed 
for  inspection.  The  expendable  materials  were  scaled  by  a  factor  of  2.4  to  account  for  the 
increase  in  number  of  blades  to  1.2  million  ( ^500*000°  ~  alternatives  were 

assumed  to  have  the  same  expendable  costs  since  they  all  rely  on  the  AFPPM  to  process 
blades.  For  the  baseline  system,  the  costs  for  the  spare  parts  were  also  scaled  by  a  factor 
of  2.4.  The  costs  for  the  spare  parts  for  alternative  1  was  assumed  to  be  roughly  half  of 
the  cost  for  the  baseline  system.  The  cost  for  spares  for  alternative  2  was  estimated  to  be 
120%  of  that  of  alternative  1  because  of  the  increased  amount  of  hardware  used.  The  cost 
for  spares  for  alternative  3  was  estimated  to  be  150%  the  cost  of  spares  for  alternative  1 
because  it  has  all  of  the  additional  hardware  as  alternative  2  replicated  and  then  additional 
hardware  to  tie  the  two  sets  together. 

The  software  development  costs  of  the  proposed  designs  depended  directly  on  the 
number  of  lines  of  code  required  to  operate  the  inspection  systems.  The  amount  of  software 
code  needed  to  be  written  for  the  alternatives  was  estimated  based  on  the  number  of  lines 
of  code  in  the  current  system.  The  number  of  lines  of  code  required  for  each  alternative 
was  entered  into  the  software  development  prediction  program  REVIC  (Revised  version  of 
intermediate  COCOMO)  (29)  to  estimate  the  cost  of  software  development.  Alternative  1 
has  no  software  development  costs.  For  alternative  2,  the  number  of  lines  of  code  needed 
was  estimated  as  25,000  ±  5000  which  is  an  estimate  of  the  number  of  lines  of  code  in 
the  baseline  system.  Since  alternative  3  has  more  information  to  process  and  control,  the 
number  of  lines  of  code  was  increased  by  10%.  The  development  costs  were  assumed  to  be 
one-time  capital  costs. 

The  capital  costs  of  hardware  were  based  on  hardware  available  to  build  the  systems 
as  proposed  in  sections  4.1,  4.2,  and  4.3.  The  hardware  development  costs  were -estimated 
to  be  the  same  as  the  the  capital  costs  of  the  new  hardware. 

The  contract  maintenance  cost  of  the  baseline  system  was  found  by  using  the  current 
maintenance  contract  for  the  IBIS  and  appropriating  one-third  of  the  cost  to  the  FPIM, 
one-third  of  the  cost  to  the  IRIM.  and  one-third  of  the  cost  to  the  AFPPM  (8).  Only  the 
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FPIM  and  the  AFPPM  costs  were  used  in  the  calculations.  The  maintenance  contract 
costs  for  the  other  three  alternatives  were  based  on  the  maintainability  values  calculated 
in  section  5.4.  The  maintainability  values  were  linearly  scaled  so  the  baseline  had  a  value 
of  one  and  the  other  values  were  a  fraction  of  the  baseline  number.  The  three  alternatives 
were  assigned  fractional  values  because  the  designs  were  more  maintainable  and  should 
have  a  lower  maintenance  cost.  The  scaled  values  were  multiplied  by  the  maintenance  cost 
of  the  current  system  to  estimate  the  maintenance  costs  for  the  alternatives. 

The  cost  of  technicians  (inspectors  and  operators)  was  calculated  by  using  an  hourly 
cost  of  $48  per  hour.  The  current  system  uses  an  average  of  3  persons  with  a  workload 
of  500,000  parts  per  year  (65).  The  labor  was  divided  into  two  parts:  (1)  the  labor  needed 
to  operate  the  washer  and  the  AFPPM,  and  (2)  the  labor  needed  to  inspect  the  blades.  The 
analysis  of  the  throughput  capacity  of  the  alternative  designs  in  section  5.6  estimated  the 
number  of  manual  inspection  hours  required  to  inspect  1.2  million  blades.  The  throughput 
estimates  only  considered  the  total  amount  of  inspection  time  and  did  not  account  for 
the  time  required  to  move  blades  into  or  out  of  the  inspection  booth,  breaks,  or  other 
non-inspection  time.  All  of  the  non-inspection  time  is  accounted  for  in  the  labor  needed  to 
operate  the  washer  and  the  AFPPM.  To  evaluate  the  alternative  designs,  it  was  assumed 
that  each  alternative  processed  1.2  million  blades  per  year,  instead  of  the  500,000  blades 
current  processed.  Since  the  number  of  blades  passing  through  the  system  increased  by  a 
factor  of  2.4,  the  cost  of  technician  labor  was  also  scaled  by  a  factor  of  2.4. 

The  EUAC  for  each  alternative  was  calculated  using  the  information  in  table  5.17. 
The  EUAC  was  converted  to  a  relative  score  in  the  range  from  0  to  10  so  the  values  could  be 
combined  with  the  scores  from  the  other  criteria  in  the  evaluation  matrix.  The  alternatives 
were  compared  to  a  best  and  a  worst  system  that  would  be  implemented.  The  best  system 
(least  expensive)  was  modeled  as  a  system  with  the  operating  costs  of  the  baseline  system, 
no  capital  costs,  and  the  personnel  costs  of  the  alternative  1  system  without  the  SQC 
personnel.  This  best  system  could  be  realized  by  removing  the  FPIIvI  and  operating  as 
the  system  operates  now.  The  worst  system  was  modeled  as  a  system  with  the  baseline 
operating  and  personnel  costs  with  the  addition  of  the  SQC  personnel,  and  150%  of  the 
capital  costs  of  alternative  3.  This  worst  system  can  be  thought  of  as  trying  to  implement 
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Criteria 

Baseline 

Alt  1 

Alt  2 

Alt  3 

EUAC  (dollars) 

1,237,603 

1,192,208 

1,351,124 

1,442.252 

Cost  Value 

7.5 

8.1 

6.0 

4.8 

Table  5.18.  Cost  Analysis 


alternative  3  with  a  cost  overrun  and  then  not  being  successful.  The  relative  ratings  for 
cost  were  calculated  for  each  system  using  equation  5.1. 


rating(x) 


{worst  -  x) 
(worst  —  best) 


X  10 


(5.1) 


The  results  are  given  in  table  5.18. 

Intuitively,  the  results  of  the  cost  analysis  seem  reasonable.  Alternative  1  is  the  least 
expensive  system  because  there  is  very  little  capital  costs  and  the  human  inspectors  are  very 
efficient  at  their  task.  Alternative  2  ranks  second  because  the  semi-automated  inspection 
system  is  initially  less  expensive  than  the  fully  automated  system  in  alternative  3.  In 
addition,  the  operating  cost  is  much  lower  for  alternative  2  than  the  baseline  because  it  is 
significantly  more  maintainable.  Alternative  3  also  ranks  higher  than  the  baseline  because 
it  is  more  maintainable  than  the  baseline  system. 

To  fully  understand  the  current  value  of  each  system,  confidence  factors  were  assigned 
to  each  alternative  to  represent  the  accuracy  of  the  information  used  in  the  cost  analysis. 
The  baseline  cost  information  was  calculate  using  information  provided  by  SA-ALC  and 
other  personnel  currently  working  with  the  system.  The  costs  were  also  scaled  to  represent 
a  projected  growth  in  the  use  of  the  system.  The  baseline  system  received  confidence 
rating  of  0.6,  because  the  data  were  pieced  together  from  several  sources  there  is  some 
uncertainty  in  the  accuracy  of  the  information. 


The  alternative  1  costs  were  developed  using  the  operation  and  maintenance  costs  of 
the  AFPPM  in  its  present  form  and  by  estimating  the  reduction  in  the  maintenance  costs 
from  removing  the  FPIM.  Overall,  the  assumptions  nsed  were  almost  the  same  as  those 
used  for  the  baseline  system,  so  alternative  1  received  the  same  confidence  rating  as  the 
baseline:  0.6. 
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Alternative  2  used  the  same  assumptions  as  alternative  1  along  with  additional  esti¬ 
mations.  The  maintenance  cost  of  the  new  hardware  was  estimated.  The  cost  of  software 
development  was  estimated  using  an  estimate  of  the  size  of  the  software  used  in  the  current 
FPIM.  The  hardware  development  costs  were  estimated  as  a  fraction  of  the  estimated  new 
hardware  costs.  In  addition,  the  inspection  performance  of  the  automated  blade  reader 
was  also  estimated.  The  confidence  rating  for  alternative  2  is  0.3,  because  there  is  a 
large  number  of  assumptions  and  there  is  some  uncertainty  in  the  accuracy  of  all  of  the 
information. 

Alternative  3  used  the  same  type  of  assumptions  and  estimations  as  alternative  2. 
Accordingly,  alternative  3  received  a  confidence  rating  of  0.3.  Table  5.19  summarizes  the 
confidence  factors  as  well  as  the  EUAC  and  utility  scores  for  each  alternative. 


Criterion 

Alternatives 

Expected  Outcome 

Confidence 

Support 

EUAC 

Value 

Level 

Rating 

Information  pieced  to- 

Baseline 

1,237,603 

C 

0.6 

gether  from  sources  in  SA- 
ALC  and  scaled  for  pro¬ 
jected  growth. 

System 

Cost 

1.  Manual 
Inspection 

1,192,208 

8.1 

C 

0.6 

Based  on  same  informa¬ 
tion  used  to  estimate  the 
baseline. 

Estimates  reduc¬ 

tion  in  maintenance  with¬ 
out  the  FPIM. 

2.  Semi- 
Automated 

1,351,124 

6.0 

LC 

0.3 

AFPPM  same  as  baseline. 
Estimate  cost  of  new  hard¬ 
ware,  software  and  hard¬ 
ware  development. 

Estimate  performance  of 

automated  system. 

3.  Fully 
Automated 

1,442,252 

4.8 

LC 

0.3 

Same  as  alternative  2. 

Table  5.19.  Summary  of  cost  evaluation. 
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5. 6  Throughput  Capacity 


To  further  compare  the  performance  of  the  three  alternatives,  the  rate  at  which 
each  inspection  system  processes  blades  was  determined.  The  average  number  of  blades 
the  system  can  process  per  hour  provides  a  measure  of  the  system’s  ability  to  handle  a 
large  surge  in  the  quantity  of  blades  requiring  inspection.  The  throughput  analysis  also 
reveals  the  number  of  hours  the  system  must  operate  each  year  and  the  amount  of  manual 
inspection  required  for  each  design  alternative.  In  addition,  the  analysis  estimated  the 
number  of  blades  humans  will  have  to  inspect  as  well  as  the  number  that  an  automated 
system  will  classify  correctly.  The  throughput  estimates  were  based  on  the  available  data 
and  the  computations  required  some  of  the  same  information  as  the  analysis  for  the  cost 
of  incorrectly  classifying  a  blade.  Appendix  K  provides  detailed  information  on  how  the 
computations  were  performed.  The  values  calculated  for  throughput  wiU  help  characterize 
the  systems’  processing  rate  and  provide  information  on  human  inspector  requirements 
for  the  annual  cost  analysis  (see  section  5.5).  In  addition,  the  upgradability  portion  of 
the  system  evaluation  uses  the  output  values  of  the  throughput  capacity  calculations  to 
estimate  the  ease  with  which  each  system  could  be  improved  in  the  future  (see  section  5.7). 

All  of  the  assumptions  listed  at  the  beginning  of  this  chapter  and  those  used  in  the 
analysis  of  classification  errors  influence  the  throughput  calculations.  These  assumptions 
describe  the  values  for  a  variety  of  design  variables  used  in  the  analysis  and  are  summarized 
in  table  3.1  at  the  beginning  of  this  chapter.  The  assumptions  about  the  design  variables 
permitted  calculations  that  would  highlight  the  differences  between  the  alternatives.  (See 
section  5.2.3  for  a  complete  list  of  assumptions.)  The  accuracy  of  the  assumptions  clearly 
influences  the  results  of  the  analysis,  so  confidence  factors  were  included  after  evaluating 
the  throughput  capacity  of  each  alternative. 

The  following  outline  provides  a  general  synopsis  of  the  calculations  for  each  alter¬ 
native.  Note  that  not  all  of  the  steps  apply  to  each  alternative. 

1.  Determine  the  number  of  blades  sent  directly  to  the  human  inspector  and  the  number 
initially  sent  to  the  automated  system. 

2.  Calculate  the  time  required  for  the  automated  portion  of  the  system  to  process 
the  blades.  Also  compute  the  number  of  blades  that  require  subsequent  manual 
inspection. 
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3.  Calculate  the  time  required  by  the  human  inspector  to  process  the  blades. 

4.  Sum  the  times  for  the  automated  portion  of  the  inspection  and  for  the  human  in¬ 
spectors  to  get  a  total  time  to  inspect  the  blades. 

5.  Divide  the  number  of  blades  inspected  by  the  total  time  to  determine  the  average 
number  of  blades  inspected  per  hour. 

The  MathCAD  program  outlined  in  appendix  K  was  used  to  perform  the  analysis 
and  table  5.20  summarizes  the  results  of  the  calculations. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Automated  hours  of  operation 

250 

0 

3333 

3333 

Manual  hours  of  inspection 

2044 

2083 

319 

0 

Total  hours  per  year 

2294 

2083 

3652 

3333 

Average  blades  per  hour 

523 

576 

329 

360 

Table  5.20.  Summary  of  throughput  calculations. 


For  the  baseline  system  and  alternatives  1  and  2,  the  number  of  hours  required 
per  year  for  human  inspectors  is  the  number  of  hours  actually  spent  inspecting.  The 
values  for  manual  inspection  in  the  table  do  not  account  for  breaks  or  other  non-inspection 
time.  Accounting  for  non-inspection  time  will  decrease  the  total  throughput  rate  of  the 
alternatives  that  require  human  inspectors.  The  cost  analysis  used  the  number  of  manual 
hours  of  inspection  shown  in  table  5.20  to  estimate  the  number  of  actual  labor  hours 
required  to  achieve  the  given  amount  of  inspection  time  (see  section  5.5). 

From  table  5.20,  the  average  number  of  blades  processed  per  hour  was  used  to  develop 
a  single  figure  of  merit  to  characterize  the  system  throughput.  Following  the  approach 
outlined  by  Athey,  the  above  results  were  compared  to  a  range  of  expected  throughput 
values.  An  exceptional  system  would  be  able  to  inspect  any  blade  type  at  a  rate  of  at 
least  600  blades  per  hour  and  regardless  of  whether  or  not  the  blade  was  defective.  An 
average  inspection  system  would  process  approximately  300  blades  per  hour,  while  a  below 
average  system  would  process  100  or  fewer  blades  per  hour.  These  values  were  based  on 
the  performance  of  the  current  system  and  the  expectations  of  the  personnel  at  SA-ALC. 
If  these  expected  throughput  values  offer  a  reasonable  standard,  then  a  figure  of  merit  can 
be  assigned  to  each  alternative  based  on  the  above  calculations. 
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The  performance  levels  of  the  baseline  system  and  alternative  1  are  very  close  to 
exceptional,  but  the  throughput  rate  of  human  inspectors  depends  on  the  type  of  blade 
and  the  percentage  of  bad  blades  in  the  population.  Some  blades  are  larger  than  others, 
which  require  more  time  for  an  inspector  to  scan.  In  addition,  defective  blades  require 
more  inspection  time  because  the  inspector  must  examine  the  suspect  region  in  detail.  A 
detailed  inspection  may  involve  looking  at  the  blade  with  a  magnifying  glass,  using  a  white 
light,  or  applying  additional  chemicals  to  highlight  the  suspect  region.  An  inspector  may 
also  wipe  the  blade  clean  and  watch  for  additional  FP  bleeding  out  (called  bleedback), 
which  would  indicate  the  presence  of  a  crack.  With  these  factors  in  mind,  the  baseline 
system  received  a  score  of  7.  Alternative  1  received  a  score  of  8,  because  it  was  slightly 
faster  than  the  baseline.  The  throughput  improvement  for  alternative  1  arises  because  no 
blades  are  processed  through  the  FPIM. 

The  remaining  alternatives  represent  average  or  slightly  below  average  performance 
in  terms  of  throughput.  Since  the  estimate  for  alternative  2  was  lower  than  that  for 
alternative  3,  alternative  2  was  given  a  score  of  4  and  alternative  3  received  a  score  of  5. 
Table  5.21  displays  the  figures  of  merit. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Blades  per  hour 

524 

576 

329 

360 

Relative  Throughput 

7 

8 

4 

5 

Table  5.21.  Relative  figures  of  merit  for  throughput. 


The  assumptions  made  above  that  allowed  for  the  throughput  estimates  seem  reason¬ 
able.  In  other  words,  the  results  make  sense  from  an  intuitive  point  of  view.  The  system 
using  only  human  inspectors  is  the  fastest,  because  the  inspectors  are  relatively  efficient 
in  all  aspects  of  the  inspection  task  and  can  adapt  to  a  variety  of  conditions.  The  baseline 
system  is  a  close  second,  because  the  human  inspectors  process  most  of  the  blades  and  the 
FPIM  only  adds  a  small  amount  of  time  to  the  total.  Alternatives  2  and  3  are  the  slowest, 
because  they  require  all  blades  to  pass  through  the  automated  inspection  system,  which 
has  a  fixed  processing  rate.  Alternative  2  is  the  worst,  because  it  requires  human  inspectors 
in  addition  to  the  automated  system,  which  reduces  overall  throughput.  Theoretically,  a 


5-27 


machine  could  be  developed  that  inspects  blades  faster  than  a  human  inspector,  but  the 
designs  for  alternatives  2  and  3  set  conservative  limits  on  the  manipulation  speed  of  the 
automated  systems. 

To  fully  characterize  the  throughput  values,  confidence  factors  for  each  alternative 
must  be  developed  to  estimate  the  accuracy  of  the  information  used  to  make  the  calcu¬ 
lations.  For  the  baseline  system,  the  throughput  calculations  were  developed  using  the 
information  provided  by  SA-ALC  management  about  human  inspector  performance  and 
the  operation  of  the  FPIM  (65).  Members  of  the  design  team  directly  observed  both  the 
manual  and  automated  inspection  processes  at  Kelly  AFB  to  confirm  the  reported  inspec¬ 
tion  times.  In  addition,  SA-ALC  provided  historical  data  describing  the  number  of  parts 
processed  through  the  AFPPM  and  the  number  of  fourth  and  ninth  stage  blades  sent  to 
the  FPIM  for  inspection.  With  all  of  this  information,  there  is  high  confidence  in  the 
throughput  estimate  for  the  baseline  system  and  it  received  a  confidence  rating  of  0.7.  The 
rating  is  not  higher,  because  the  throughput  calculations  did  not  account  for  the  variation 
in  manual  inspection  time  for  different  types  of  blades  (large  blades  require  more  time). 

Design  alternative  1  is  very  similar  to  the  baseline  system,  except  that  there  is  no 
automated  inspection.  The  human  inspectors  should  be  able  to  perform  at  the  same  rate 
for  this  alternative  as  they  did  in  the  baseline  system.  The  information  used  to  justify  the 
confidence  level  in  the  baseline  also  applies  to  this  alternative,  so  alternative  1  received  a 
confidence  factor  of  0.7,  as  well. 

Alternative  2  requires  an  estimate  of  the  performance  of  the  semi-automated  blade 
inspection  sy..  em.  A  robotic  simulation  using  the  IGRIP  software  (see  appendix  F)  demon¬ 
strated  that  one  system  configuration  could  process  blades  at  an  average  of  less  than  10 
seconds  per  blade.  Further  research  and  discussions  with  commercial  vendors  indicated 
that  the  time  required  to  capture  and  process  the  images  would  be  insignificant  compared 
to  the  manipulation  time.  In  addition,  the  image  processing  and  classification  software 
model  Wcis  used  to  estimate  the  expected  false  call  rate  of  the  semi -automated  system 
(see  appendix  G).  These  models  provide  reasonable  estimates  of  the  system  performance, 
but  the  operation  times  may  differ  in  the  actual  system.  The  human  inspectors  should 
maintain  the  same  level  of  performance,  with  the  same  confidence,  as  in  the  previous  al- 
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ternatives.  Since  alternative  2  required  models  to  estimate  the  throughput  performance, 
the  confidence  in  the  information  is  slightly  lower  than  the  baseline  system  and  alterna¬ 
tive  1.  Alternative  2  received  a  confidence  factor  of  0.6,  which  indicates  the  information  is 
relatively  accurate,  but  additional  work  is  required  for  complete  verification. 

Alternative  3  assumed  the  system  could  automatically  process  all  blades  at  the  same 
rate  as  the  system  in  alternative  2.  There  was  no  additional  robotic  simulation  or  software 
modeling  for  this  specific  configuration.  The  values  used  to  calculate  throughput  were 
developed  as  the  best  estimates  of  how  the  system  could  perform.  Accordingly,  there  is 
little  confidence  in  the  accuracy  of  the  figures  and  this  alternative  received  a  confidence 
factor  of  0.3.  The  low  confidence  rafug  is  a  clear  signal  that  this  alternative  must  be 
explored  in  greater  depth  to  fully  characterize  its  performance.  Table  5.22  displays  the 
utility  scores  and  the  confidence  ratings  for  each  alternative. 

Table  5.22  provides  a  complete  summary  of  the  throughput  analysis.  The  utility 
scores  along  with  the  confidence  factors  allow  a  distinction  to  be  made  between  the  baiseline 
system  and  the  alternative  designs.  These  values  serve  as  an  input  to  the  final  evaluation 
matrix  at  the  end  of  the  chapter. 

5. 7  Upgradability 

The  final  criterion  used  to  evaluate  the  alternatives  is  upgradability.  The  upgrad¬ 
ability  of  a  system  can  be  defined  as  the  ability  to  improve  the  quality  of  its  performance 
or  the  ability  to  extend  its  usefulness.  We  can  judge  the  upgradability  of  the  alternatives 
by  the  number  of  interconnections  with  other  components,  the  complexity  of  the  interac¬ 
tions  with  other  components,  and  the  flexibility  to  adapt  to  changing  requirements  of  the 
system.  Judging  the  upgradability  of  up  ieveloped  systems  is  inherently  qualitative.  In  an 
effort  to  reduce  inconsistencies  in  judgement,  four  likely  upgrade  scenarios  were  developed 
to  judge  each  alternative.  The  most  likely  upgrade  scenarios  for  the  system  are: 

1.  Increase  system  throughput  (greater  than  1,200,000  blades  per  year), 

2.  Increase  the  number  of  engine  types  that  the  system  can  inspect  (compressor- type 

blades  longer  than  10  inches), 

3.  Increase  the  types  of  blades  the  system  can  inspect  to  include  turbine  blades,  and 
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Criterion 

Alternatives 

Expected  Outcome 

Confidence 

Support 

Blades/Hour 

Value 

Level 

Rating 

■ 

SA-ALC 

data  about  human  inspec¬ 
tor  performance. 

Baseline 

524 

1 

C 

Historical  data  from  1991 

about  AFPPM  and  FPIM: 

blades  processed  and  per¬ 
centage  of  false  calls. 

System 

Through¬ 

put 

1.  Manual 
Inspection 

576 

8 

C 

Based  on  same  informa¬ 
tion  used  to  estimate  the 
baseline. 

Assumes  huma  perfor¬ 
mance  remains  con*  ant 

without  the  FPIM. 

2.  Semi- 
Automated 

329 

1 

C 

0.6 

Robot  simulation  to  es¬ 
timate  the  manipulation 
time. 

Image  processing  and  clas¬ 
sification  software  model 
to  estimate  error  rate  per¬ 
formance. 

Hiunan 

1 

inspectors  maintain  previ¬ 
ous  performance. 

3.  Fully 
Automated 

360 

5 

LC 

0.3 

Manipulator  performjince 
equivalent  to  edtemative  2. 
Assumes  processing  speed 
does  not  change  for  differ¬ 
ent  blade  types. 

_ 

_ 1 

No  additional  simulation. 

Table  5.22.  Summary  of  throughput  evaluation. 
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4.  Increase  the  ability  of  the  system  to  categorize  blades  as  cracked  (reduce  the  false 
call  rate). 


Each  alternative  will  be  evaluated  to  determine  to  what  degree  it  can  be  upgraded  for 
each  of  the  projected  scenarios  while  maintaining  performance  constraints.  For  this  com¬ 
parison,  we  will  establish  levels  of  upgradability  in  accordance  with  Athey’s  approach  (2). 
For  each  scenario,  the  system  is  analyzed  to  determine  if  it  meets  one  of  the  following 
descriptions: 

1.  not  upgradable  without  total  redesign  or  extensive  rework  (barely  acceptable  =  2), 

2.  major  modifications  or  changes  required  to  current  system  as  indicated  by  several 
component  replacements  in  addition  to  changes  to  other  components  (below  average 
=  4), 

3.  minor  additions  to  present  system  required  such  as  modular  parts  replacement  or 
add-ons  (average  =  6), 

4.  minor  changes  to  present  system  required  such  as  software  changes  or  adjustments 
to  components  (above  average  =  8),  or 

5.  no  alterations  required  (exceptional  =  10). 

After  judging  each  alternative  against  the  four  upgrade  scenarios,  the  scores  will  be 
averaged. 

The  current  system  can  be  upgraded  to  increase  the  number  of  engine  and  blade 
types  that  the  system  can  inspect  since  human  inspectors  are  part  of  the  system.  Since  the 
current  system  relies  heavily  on  human  inspectors,  it  can  readily  adapt  to  changes  in  the  size 
as  well  as  the  types  of  blades  it  must  inspect  with  no  alterations.  Additionally,  by  simply 
reducing  the  FPIM’s  workload,  the  false  call  rate  of  the  system  will  be  upgraded.  However, 
the  current  system  cannot  increase  its  throughput  and  maintain  an  acceptable  level  of 
performance  without  some  modifications.  If  the  human  inspectors  are  overtaxed  because 
of  an  increased  workload  or  long  workday,  their  performance  will  suffer.  Their  ability  to 
properly  classify  blades  will  decrease  as  their  work  rate  or  their  workday  increases.  The 
only  way  to  successfully  increase  the  throughput  of  the  current  system  is  to  add  inspectors. 
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The  current  system  receives  an  average  upgradability  score  of  8.5.  This  score  may  be 
somewhat  misleading.  The  system  as  a  whole  is  upgradable,  but  if  constraints  are  placed  on 
the  percentage  of  blades  the  FPIM  must  inspect,  the  results  change.  The  FPIM  is  capable 
of  meeting  the  throughput  requirements  of  upgrade  scenario  1  with  no  alterations,  but  it 
is  not  upgradable  to  inspect  other  blade  types  or  to  reduce  the  false  call  rate.  Accounting 
for  the  FPIM’s  inflexibility,  would  give  it  an  average  score  of  4.  However,  since  the  manual 
inspectors  are  a  significant  part  of  the  system,  the  upgrade  score  for  the  entire  system 
remains  8.5. 

Alternative  1  is  very  similar  to  the  current  system  in  terms  of  upgradability  since 
it  relies  on  the  human  inspectors.  It  receives  an  average  score  of  9  for  many  of  the  same 
reasons  as  the  current  system.  The  average  score  for  alternative  1  is  slightly  higher  since 
it  does  not  require  changes  to  compensate  for  the  FPIM’s  false  call  rate. 

Alternative  2  is  capable  of  meeting  the  upgrade  requirements  for  increased  through¬ 
put,  increased  number  of  engine  types,  and  increased  number  of  blade  types  with  minor 
changes.  In  order  for  the  throughput  of  the  system  to  be  increased,  the  system  will  have 
to  be  altered  to  be  operated  with  additional  shifts.  For  the  system  to  be  able  to  inspect 
different  engine  types,  minor  changes  to  the  classification  algorithm  would  have  to  be  made 
to  account  for  multiple  images  for  one  side  of  the  blade.  Also,  in  order  for  the  system  to  be 
able  to  inspect  other  blade  types,  the  algorithm  would  have  to  be  changed  to  account  for 
the  difference  in  blade  surface  of  turbine  blades.  The  system  is  also  upgradable  to  reduce 
the  false  call  rate  with  major  modifications  or  changes  to  the  current  system.  The  major 
modifications  or  changes  for  reducing  the  false  call  rate  involve  using  perhaps  a  different 
sensor,  or  multiple  images  using  the  same  sensor,  to  collect  additional  features  that  could 
be  used  to  more  accurately  classify  blades.  Alternative  2  receives  an  average  score  of  8. 

Alternative  3  is  already  able  to  meet  the  increased  number  of  blade  types  require¬ 
ments  of  upgrade  scenario  4  with  no  alterations.  Additionally,  alternative  3  can  be  up¬ 
graded  with  minor  changes  to  the  manipulation  plan  and  the  classification  software  to 
inspect  blades  longer  than  10  inches,  and  to  the  hours  of  operation  to  increase  the  system 
throughput.  It  may  also  be  capable  of  meeting  the  requirement  for  inspecting  other  blade 
types  with  minor  changes.  The  minor  changes  all  pertain  to  the  classification  algorithm. 
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The  algorithm  would  have  to  be  written  to  account  for  the  difference  in  the  blade  surface 
of  turbine  blades.  Alternative  3  receives  an  average  score  of  9.  Table  5.23  summarizes  the 
initial  results. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Throughput 

6 

6 

8 

8 

Engine  Types 

10 

10 

8 

8 

Blade  Types 

10 

10 

8 

8 

False  Calls 

8 

10 

4 

10 

Average  Score 

8.5 

9 

7 

8.5 

Table  5.23.  Summary  of  upgradability  values 


In  estimating  the  upgradability  of  the  different  systems,  there  is  a  degree  of  uncer¬ 
tainty  associated  with  each  estimate.  A  way  to  reflect  the  uncertainty  of  the  assigned 
relative  values  is  to  evaluate  the  source  of  the  data  used  to  analyze  each  alternative.  Each 
alternative’s  relative  score  will  be  multiplied  by  a  number  between  0  and  1  to  reflect  the 
degree  of  confidence  in  the  data.  For  the  current  system  and  alternative  1,  the  data  used  to 
evaluate  their  degree  of  upgradability  came  from  historical  data  as  well  as  interviews  with 
users  and  the  original  system  designers.  There  is  high  confidence  in  the  estimates,  but  they 
are  still  based  on  estimates  of  future  performance,  thus  they  received  a  confidence  value 
of  0.7.  The  support  for  the  estimates  of  the  degree  of  upgradability  of  alternatives  2  and  3 
come  from  estimations  and  simulations  based  on  data  from  experiments.  The  confidence 
in  these  estimates  is  low  and  are  assigned  a  value  of  0.4.  Table  5.24  summarizes  the  results. 

The  upgradability  analysis  indicates  that  all  of  the  systems  are  at  least  above  average 
for  upgradability.  This  comes  as  little  surprise  when  you  consider  the  flexibility  of  human 
inspectors  to  adapt  to  changing  requirements  and  that  all  the  alternatives  were  designed  to 
meet  the  constraint  that  all  alternatives  be  upgradable.  However,  alternative  1  is  judged  to 
be  most  upgradable  followed  by  alternative  3  and  the  current  system,  and  least  upgradable 
is  alternative  2. 
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Criterion 

Alternatives 

Expected  Outcome 

Confidence 

Support 

Value 

Level 

Rating 

Baseline 

8.5 

C 

Historical  data  on  human 
inspection. 

Interviews  with  FPIM 

users  and  original  system 

designers. 

Upgrad¬ 

ability 

1.  Manual 
Inspection 

9 

C 

Based  on  same  informa¬ 
tion  used  to  estimate  the 
baseline. 

Assumes  hiunan  perfor¬ 
mance  remains  constant 

without  the  FPIM. 

2.  Semi- 
Automated 

7 

C 

Robotic  simulation  of 
workcell. 

Image  processing  and  clas¬ 
sification  software  model 
to  estimate  error  rate  per¬ 
formance. 

Human 

inspectors  maintain  previ¬ 
ous  performance. 

3.  Fully 
Automated 

8.5 

LC 

0.4 

Manipulator  performance 
equivalent  to  alternative  2. 

No  additional  simulation. 

Table  5.24.  Summary  of  upgradability  evaluation. 
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5.8  Evaluation  Matrix 


Having  analyzed  the  five  criteria  in  detail,  it  is  now  possible  to  complete  the  evalu¬ 
ation  process  and  to  identify  the  system  that  provides  the  best  solution  to  the  inspection 
problem.  The  evaluation  matrix,  shown  as  table  5.25,  represents  a  summary  of  all  of  the 
information  developed  in  the  preceding  five  sections.  The  matrix  also  completes  the  anal¬ 
ysis  by  showing  the  results  of  applying  the  confidence  factors  and  computing  the  total  and 
discounted  utilities  for  each  alternative. 

The  five  high  level  criteria  are  listed  on  the  left  side  of  the  matrix  along  with  their 
corresponding  relative  weights,  which  were  calculated  with  the  preference  chart.  Each 
alternative  lists  its  raw  utility  values  and  confidence  factors  under  the  columns  labeled 
“R”  and  “C”,  respectively.  The  total  utility,  located  in  column  “U”,  is  the  product  of  the 
criterion’s  relative  weight  and  the  raw  utility  score.  The  discounted  utility  is  the  product 
of  the  total  utility  and  the  confidence  factor  and  is  listed  in  column  “D”.  For  example,  the 
total  utility  of  the  blade  misclassification  criterion  and  alternative  2  is  292,  because  the 
weighting  factor  is  40.0  and  the  raw  utility  score  is  7.3,  (40.0  x  7.3  =  292).  The  discounted 
utility  for  the  same  alternative  and  criterion  is  146,  because  the  confidence  factor  is  0.5, 
(0.5  X  292  =  146). 

After  all  of  the  total  and  discounted  utility  scores  are  computed,  the  values  are 
summed  for  each  alternative  to  get  a  total  value  and  a  total  discounted  value,  which  are 
shown  in  the  bottom  row  of  the  matrix.  The  overall  confidence  level,  also  shown  at  the 
bottom  of  the  matrix,  is  the  result  of  dividing  the  total  discounted  value  by  the  total  value 
and  represents  a  weighted  average  of  the  confidence  levels  associated  with  each  alternative. 

To  interpret  the  results  of  the  evaluation  matrix,  the  total  utility,  the  total  discounted 
utility,  and  the  overall  confidence  in  the  final  row  must  be  considered  in  unison.  Each  value 
provides  a  distinct  type  of  information  about  the  relative  performance  of  each  alternative. 

The  total  utility  describes  the  expected  performance  of  the  alternative  when  all 
of  the  underlying  assumptions  associated  with  the  alternative  are  true.  Scanning  the 
bottom  section  of  the  evaluation  matrix  reveals  that  alternative  3,  the  fully  automated 
inspection  system,  offers  the  best  potential  performance  with  a  score  of  637.0.  Alternative  3 
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scored  reasonably  well  in  all  categories,  but  dominated  the  criterion  measuring  the  cost 
of  inspection  errors.  This  criterion  was  the  most  important  and,  accordingly,  had  the 
largest  weighting  factor  which  gave  alternative  3  the  best  performance.  Alternatives  2 
and  1  had  the  next  lowest  scores  in  terms  of  total  utility,  so  they  would  rank  second  and 
third,  respectively,  behind  alternative  3.  The  baseline  system  had  the  lowest  score  for  toted 
utility  and  would  be  the  least  desirable  alternative  among  the  ones  examined.  The  baseline 
system  received  relatively  low  scores  for  its  maintainability  and  its  cost,  which  caused  it 
to  have  a  significantly  lower  expected  performance  than  the  other  design  options. 

The  total  utility  scores  do  not  account  for  the  certainty  of  the  information  used  in 
the  evaluation  process,  so  the  total  discounted  utility  must  be  considered  as  well.  When 
the  discount  factors  are  applied,  the  preference  order  for  the  solutions  changes  consider¬ 
ably.  Alternative  1  has  the  highest  total  discounted  utility,  which  means,  given  all  of  the 
information  available  during  the  study,  it  is  the  best  choice.  The  discounted  utility  values 
for  alternative  1  show  that  it  received  consistently  high  scores  across  all  criteria.  The 
alternative  2  system  received  the  second  highest  value  for  total  discounted  utility,  followed 
by  the  baseline  system  and,  finally,  alternative  3.  Even  with  high  confidence  factors,  the 
baseline  could  not  compensate  for  its  low  raw  utility  score  for  the  maintainability  and  cost 
criteria.  Alternative  3  received  the  lowest  discounted  score  because  of  its  low  confidence 
values.  Without  additional  research,  the  best  course  of  action  for  SA-ALC  at  this  time  is 
to  pursue  the  design  described  in  alternative  1. 

Having  considered  total  utility  and  total  discounted  utility,  the  overall  confidence 
factors  must  now  be  addressed.  As  the  final  piece  of  the  evaluation  process,  the  overall 
confidence  factors  estimate  the  accuracy  of  all  information  associated  with  a  particular 
design  alternative.  Alternative  1  received  the  highest  overall  confidence  factor,-with  a  value 
of  0.68,  and  the  baseline  system  was  a  close  second  with  a  value  of  0.67.  Alternatives  2 
and  3  received  the  significantly  lower  scores  of  0.48  and  0.43,  respectively.  These  confidence 
factors  are  completely  understandable  considering  the  supporting  information  associated 
with  each  alternative.  The  information  for  the  baseline  system  is  well  defined  since  the 
system  can  be  observed  directly  and  some  historical  data  was  available  for  the  analysis. 
The  overall  confidence  factor  for  the  baseline  is  not  higher  for  two  primary  reasons.  First, 


5-37 


the  evaluation  process  still  required  a  variety  of  assumptions  and  approximations  about 
the  system  in  order  to  quantify  its  performance.  Second,  evidence  that  supports  the 
performance  capability  is  lacking  due  to  the  absence  of  an  SQC  plan.  Alternative  1  was  very 
similar  to  the  baseline  system  and,  therefore,  received  a  similar  score  for  confidence.  Design 
alternatives  2  and  3  were  based  on  experiments  and  models  created  to  demonstrate  portions 
of  the  required  technology.  The  actual  performance  of  these  systems  in  an  industrial 
environment  still  contains  some  level  of  uncertainty.  The  need  for  more  experience  and 
additional  research  with  the  technology  required  for  alternatives  2  and  3  causes  the  overall 
confidence  factors  for  these  two  alternatives  to  be  lower  than  those  for  the  baseline  and 
alternative  1. 

In  addition  to  the  accuracy  of  the  supporting  information  for  the  design  alternatives, 
the  overall  confidence  factors  indicate  the  amount  of  research  required  to  fully  develop  the 
corresponding  design.  Alternative  1  can  be  implemented  directly  with  minimal  research, 
since  it  closely  parallels  the  baseline  system.  Alternatives  2  and  3  require  considerable 
research  to  further  refine  the  designs  and  fuUy  justify  some  of  the  underlying  assumptions 
made  during  the  analysis.  The  models  created  in  support  of  alternative  2  are  a  good  first 
step,  but  they  are  not  sufficient  to  move  directly  to  a  production  system.  The  design 
for  alternative  2  could  be  fully  developed  and  implemented  in  approximately  three  years. 
Alternative  3  requires  additional  research  to  verify  the  design,  but  it  offers  the  potential 
for  dramatic  improvements  in  the  quality  of  the  inspection  process.  It  is  likely  that  the 
alternative  3  design  will  require  at  least  five  years  of  additional  research  and  development 
before  it  is  operational. 

5.9  Sensitivity  Analysis 

The  system  evaluation  relied  on  Athey’s  approach  to  establish  the  weights  for  each 
criterion  and  to  quantify  the  value  of  each  alternative  in  terms  of  a  common  utility  mea¬ 
sure.  The  final  results  of  the  evaluation  are  clearly  sensitive  to  the  weights  produced  by 
the  preference  chart  and  the  utility  values  derived  from  the  original  performance  measures, 
which  included  dollar  cost  and  inspection  rate.  This  section  briefly  examines  how  changing 
the  preference  chart  influences  the  final  results.  In  addition,  the  evaluation  will  be  com- 
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pleted  using  only  the  dollar  cost  of  the  alternatives  and  show  that  the  results  remain  the 
same  even  with  these  changes  to  the  evaluation  process. 


5.9.1  Preference  Chart  Sensitivity.  Appendix  L  presents  a  full  discussion  of  how  a 
change  in  the  preference  charts  influences  the  flnal  evaluation  results.  The  critical  results 
are  summarized  here.  The  original  chart  was  discussed  in  section  5.2.1  and  based  the 
weights  on  the  perspective  that  the  cost  of  incorrectly  classifying  blades  was  the  most 
important  criteria.  The  revised  preference  chart  is  bcised  on  the  stance  that  maintainability 
is  more  important  than  the  cost  of  misclassification  errors.  The  other  three  criteria  did  not 
change  their  relative  rankings.  The  new  weights  for  the  criteria  are  shown  in  table  5.26. 


Criteria 

Weight 

Misclassification 

22.5 

Maintainability 

40.0 

Annual  Cost 

17.5 

Throughput 

12.5 

Upgradability 

7.5 

Table  5.26.  Alternative  weights  for  preference  chart  sensitivity  analysis. 


Using  the  weights  in  table  5.26  and  the  utility  values  developed  for  the  original 
evaluation  causes  some  minor  changes  in  the  results  of  the  new  evaluation  matrix.  The 
weights  are  completely  independent  of  the  utility  scores,  so  changing  one  type  of  value  does 
not  influence  the  other.  The  critical  values  for  the  new  evaluation  are  the  total  values  and 
the  total  discounted  values,  which  are  shown  in  table  5.27. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Total  Value  (U) 

437.3 

561.8 

560.0 

563.5 

Total  Discounted  Value  (D) 

299.9 

366.0 

278.3 

229.7 

Table  5.27.  Results  of  the  evaluation  with  new  criteria  weights. 


The  new  evaluation  does  not  change  the  final  results  significantly.  Alternative  3  still 
has  the  highest  score  in  terms  of  total  value  and  it  is  followed  closely  by  alternatives  1 
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and  2.  For  the  total  discounted  values,  alternative  1  remains  at  the  top  and  the  baseline 
system  is  in  second  place. 


Changing  the  weights  generated  by  the  preference  chart  to  reflect  a  different  perspec¬ 
tive  on  the  importance  of  the  criteria  did  not  change  the  final  results  of  the  evaluation. 
Appendix  L  provides  a  more  complete  description  of  the  preference  chart  sensitivity  anal¬ 
ysis. 


5.9.2  An  Alternate  Evaluation  Approach.  Another  method  to  evaluate  the  alter¬ 
natives  is  to  measure  each  design  strictly  in  terms  of  monetary  value.  If  the  criteria  were 
measured  in  terms  of  dollars,  the  total  value  of  each  alternative  could  be  compared  directly 
without  converting  to  utility  values  or  using  confidence  factors.  Appendix  M  demonstrates 
how  to  evaluate  the  alternatives  solely  in  terms  of  dollar  cost. 

Table  5.28  presents  the  CICB  and  EUAC  final  cost  amounts  as  well  as  the  cost 
avoidance  amounts.  The  cost  avoidance  is  the  amount  of  money  saved  by  the  Air  Force  for 
implementing  one  of  the  three  alternatives.  The  cost  avoidance  is  calculated  by  subtracting 
the  total  cost  associated  with  each  of  the  three  alternatives  from  the  total  cost  associated 
with  the  baseline  system.  This  secondary  evaluation  verifies  the  original  evaluation  and 
confirms  that  in  the  long  term,  alternative  3  offers  the  best  potential  to  improve  the  blade 
inspection  system. 


CICB($) 

EUAC($) 

Total  Cost($) 

Cost  Avoidance($) 

Baseline 

94.90  M 

1.238  M 

96.14  M 

0 

Alt  1 

95.03  M 

1.192  M 

96.22  M 

-0.08  M 

Alt  2 

36.08  M 

1.351  M 

37.43  M 

58.71  M 

Alt  3 

24.46  M 
_ 1 

1.442  M 

25.90  M 

70.24  M 

Table  5.28.  Cost  Analysis 


The  two  forms  of  sensitivity  analysis  discussed  above  show  that  the  results  of  the 


original  evaluation  are  not  sensitive  to  minor  changes  in  the  system  evaluation.  The  anal¬ 
ysis  provides  evidence  that  the  evaluation  process  is  sound  and  that  the  final  results  are 


robust. 
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5.10  Summary 

The  evaluation  process  has  combined  all  of  the  information  developed  in  the  preceding 
chapters  to  determine  the  best  course  of  action  for  SA-ALC  to  improve  their  inspection 
process.  The  analysis  required  a  variety  of  assumptions  and  approximations  in  order  to 
quantify  the  performance  of  each  design  alternative  in  a  meaningful  way.  The  evaluation 
process  also  developed  an  interaction  table  outlining  the  relationship  between  the  critical 
design  variables  and  the  five  major  criteria.  The  design  variable  table  served  as  a  guide 
on  how  to  update  the  analysis  when  one  of  the  design  variables  changed,  which  made  the 
design  process  more  efficient. 

The  evaluation  process  utilized  a  systematic  approach  to  analyze  each  design  with 
respect  to  the  criteria  and  to  determine  which  alternative  offered  the  highest  potential  to 
improve  the  jet  engine  blade  inspection  process  at  SA-ALC.  The  results  indicated  that  the 
best  course  of  action  for  the  short  term  would  be  to  pursue  design  alternative  1,  which 
recommends  removing  the  FPIM  from  service  and  using  only  manual  inspection  to  process 
the  blades.  This  design  also  requires  the  development  of  statistical  quality  control  methods 
to  help  characterize  the  system’s  performance  on  a  regular  basis  and  to  identify  additional 
areas  for  improvement.  Alternative  1  is  superior  to  the  baseline  system,  because  the  new 
design  offers  significant  improvements  in  terms  of  system  maintainability  and  it  lowers 
the  inspection  cost  without  compromising  the  system’s  ability  to  detect  cracked  blades. 
Alternative  1  had  the  highest  total  discounted  value  with  a  score  of  338.7.  The  value  for 
alternative  1  was  14%  larger  than  the  value  for  alternative  2,  which  was  the  next  closest 
score.  In  addition,  alternative  1  outperformed  the  other  alternatives  by  having  consistently 
high  confidence  factors  (.68  for  alternative  1,  compared  to  .48  and  .43  for  alternatives  2 
and  3,)  and  by  delivering  reasonable  performance  for  each  criterion. 

Even  though  alternative  1  received  the  highest  evaluation  score,  the  results  of  the 
evaluation  matrix  indicated  that  alternatives  2  and  3  were  also  worthy  of  further  consid¬ 
eration.  Alternative  2  had  a  total  utility  score  that  was  17%  higher  than  the  score  for 
alternative  1  and  alternative  3  had  a  total  utility  score  that  was  superior  to  all  of  the 
others.  The  high  utility  scores  mean  alternatives  2  and  3  could  offer  even  better  inspec¬ 
tion  performance  than  alternative  1.  The  only  obstacle  to  implementing  the  automated 
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inspection  systems  is  the  amount  of  research  required  to  fully  develop  the  more  advanced 
designs.  Alternatives  2  and  3  would  respectively  need  approximately  three  and  five  years 
of  additional  research  and  development  before  they  could  become  operational. 

It  is  important  to  note  that  the  final  results  show  the  expected  relatwe  performance 
of  the  alternatives  in  solving  the  inspection  problem.  Analyzing  the  relative  performance 
allows  the  superior  alternative  to  be  identified,  even  when  the  information  used  in  the 
analysis  is  not  perfect. 


VI.  Conclusions  and  Recommendations 


6. 1  Summary 

Jet  engine  blades  are  subjected  to  high  stresses  during  operation.  A  cracked  blade 
in  a  Jet  engine  could  reduce  performance,  fracture  and  cause  the  engine  to  malfunction,  or 
fracture  and  cause  complete  engine  failure.  Due  to  the  seriousness  of  the  consequences  of 
blade  failures,  the  Air  Force  places  a  high  priority  on  the  proper  inspection  of  the  blades. 
As  routine  preventive  maintenance  for  jet  engines,  the  blades  are  removed  from  the  core 
and  inspected  for  cracks.  Currently,  the  San  Antonio  Air  Logistics  Command  Jet  Engine 
Division  (SA-ALC/LP)  uses  the  Integrated  Blade  Inspection  System  (IBIS)  to  perform 
nondestructive  inspection  for  jet  engine  blades  using  fluorescent  penetrant  (FP).  The  IBIS 
offers  improvements  over  manual  blade  preparation  and  inspection,  but  SA-ALC  believes 
tiie  system  can  be  further  enhanced  with  appropriate  modifications.  The  objective  of 
the  systems  engineering  design  study  was  to  identify  alternatives  that  could  improve  the 
performance  of  the  blade  inspection  system.  The  study  identifies  three  alternatives  which 
could  potentially  improve  the  present  system.  All  three  alternative  designs  incorporate  the 
Automated  Fluorescent  Penetrant  Preprocessing  Module  (AFPPM)  and  require  statistical 
quality  control  (SQC)  techniques  be  applied  to  measure  the  overall  performance  of  the 
system  and  to  help  maintain  control  over  the  process.  The  SQC  plan  will  also  help  identify 
ways  to  improve  the  performance  of  the  IBIS  process. 

The  first  design  option  relies  solely  on  manual  inspection  for  all  of  the  blades  and 
calls  for  the  removal  of  the  Fluorescent  Penetrant  Inspection  Module  (FPIM).  The  main 
advantage  of  alternative  1  is  that  it  can  be  implemented  quickly  and  eaisily  for  a  significant 
operating  cost  savings  and  reasonable  overall  performance. 

The  second  alternative  is  a  semi-automated  blade  inspection  system  that  would  op¬ 
erate  in  series  with  human  inspectors.  The  system  would  screen  all  blades  and  remove 
the  good  and  the  cracked  blades,  sending  the  indeterminate  blades  to  the  inspectors.  The 
human  inspectors  would  still  be  an  integral  part  of  the  inspection  system,  but  their  work¬ 
load  would  be  significantly  reduced  to  approximately  25  percent  of  the  total  blades.  The 
overall  inspection  consistency  would  improve  as  a  result  of  the  addition  of  an  automated 
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blade  reader.  This  design  offers  the  advantages  of  improved  crack  detection  performance 
and  serves  as  a  stepping-stone  to  the  next  generation  system. 

The  final  design  alternative  is  a  fully  automated  inspection  system  capable  of  cor¬ 
rectly  classifying  all  blades  without  human  inspectors.  This  design  relies  heavily  on  the 
expected  improvements  in  classification  ability  afforded  by  the  use  of  a  neural  network 
that  uses  additional  features  provided  by  an  image  subtraction  technique.  The  system 
requires  extensive  technical  research  before  it  can  be  implemented.  Since  this  option  does 
not  require  human  inspectors,  it  offers  the  advantage  of  consistent  performance  over  long 
periods  of  time. 

The  design  study  developed  several  models  to  evaluate  each  design  alternative  and 
compare  them  to  the  baseline  system.  The  models  estimated  the  impact  of  inspection 
errors,  maintainability,  expected  equivalent  uniform  annual  costs,  system  throughput,  and 
the  potential  upgradability  of  each  design.  In  addition,  a  robotic  simulation  and  software 
image  processing  and  classification  were  developed  to  validate  the  alternative  2  design. 
Charge  coupled  device  (CCD)  camera  experiments  were  also  performed  to  identify  new 
methods  for  collecting  data  and  detecting  crack  indications.  Athey’s  evaluation  technique 
was  applied  to  quantify  the  results  of  all  models  with  a  consistent  notation  (2).  The  final 
results  of  the  decision  analysis  indicate  the  progressive  steps  required  to  solve  the  blade 
inspection  problem. 

6.2  Inspection  System  Recommendations 

To  improve  the  blade  inspection  process,  SA-ALC  should  take  the  following  series 
of  actions.  The  FPIM  should  be  removed  from  service,  because  it  does  not  significantly 
improve  the  inspection  capability,  it  has  no  foreseeable  room  for  growth,  and  it  is  expensive 
to  maintain.  Blade  inspection  should  continue  with  human  inspectors  with  the  addition 
of  an  SQC  plan  to  increase  the  control  over  the  performance  of  the  IBIS  process  and  to 
identify  ways  to  enhance  the  system’s  performance.  The  AFPPM  should  be  maintained 
and  additional  experiments  conducted  to  identify  the  significant  variables  which  influence 
the  quality  of  the  prepared  blades.  Research  should  continue  to  further  develop  the  design 
for  the  next  generation  automated  blade  inspection  system. 
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The  semi-automated  system  could  be  developed,  tested,  and  operational  within  the 
next  three  to  five  years.  The  technology  is  available  and  only  a  small  amount  of  additional 
research  would  be  necessary  to  complete  the  design.  The  expected  inspection  performance 
of  the  semi-automated  system  is  not  as  high  as  that  for  the  fully  automated  system,  because 
the  semi-automated  system  still  relies  on  human  inspectors  for  part  of  the  task.  However, 
upgrading  the  semi-automated  system  to  the  fully  automated  system  would  be  a  logical 
step  in  moving  toward  the  more  advanced  design. 

The  primary  objective  should  be  to  continually  move  in  the  direction  of  a  fully  au¬ 
tomated  system.  At  this  time,  a  fully  automated  inspection  system  is  a  viable  alternative, 
but  there  is  lower  confidence  in  the  accuracy  of  this  design  than  in  the  other  alternatives. 
The  fully  automated  system  is  feasible  if  SA-ALC  requires  the  capability  and  chooses  to 
expend  the  resources  for  additional  research  and  development.  A  more  conservative  ap¬ 
proach  would  be  to  implement  a  semi-automated  system  with  the  goal  of  evolving  it  into 
a  fully  automated  system  as  research  progresses.  This  final  design  alternative  offers  the 
best  potential  to  improve  the  performance  of  the  overall  inspection  process. 

6.3  Study  Contributions 

This  study  made  several  noteworthy  contributions  by; 


•  identifying  the  need  for  and  prescribing  a  statistical  quality  control  plan  for  the 
inspection  process 

•  developing  classification  software  to  validate  the  concept  of  using  neural  networks  to 
classify  blades  more  accurately 

•  demonstrating  the  potential  advantages  of  using  charge-coupled  device  (CCD)  cam¬ 
eras  for  data  gathering 

•  providing  a  means  to  quantify  a  portion  of  the  cost  of  incorrectly  classifying  jet  engine 
blades 

•  identifying  a  method  of  using  multiple  images  to  more  accurately  classify  blades 

Continued  research  should  focus  on  improving  or  expanding  the  usefulness  of  these 
contributions  to  provide  more  accurate  information  for  use  in  developing  future  fluorescent 
penetrant  inspection  systems. 
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6-4  Recommendations  for  Future  Research 

The  design  study  has  revealed  several  areas  that  deserve  additional  research  beyond 
the  scope  of  this  project. 

1.  The  performance  of  an  inspection  system  should  be  monitored  to  aid  in  identify¬ 
ing  when  the  system  is  not  functioning  correctly  and  identifying  potential  areas  for 
improvement.  An  appropriately  specified  statistical  quality  control  (SQC)  plan  can 
help  do  both.  Research  should  be  aimed  at  developing  a  detailed  SQC  plan  for  the 
IBIS  process  and  quantifying  the  impact  of  implementing  the  plan. 

2.  The  image  processing  and  classification  software  should  be  improved  to  enhance  the 
performance  of  any  automated  inspection  system.  The  research  should  focus  on 
collecting  better  data  about  FP  indications,  extracting  different  features  from  each 
indication,  and  developing  the  neural  network  procedures  for  blade  classification. 
The  software  could  also  classify  the  blades  into  more  than  two  categories  to  further 
enhance  the  overall  inspection  performance.  Appendix  G  contains  a  complete  list  of 
recommendations  on  how  to  improve  the  software. 

3.  Optimized  fluorescent  penetrant  preprocessing  for  an  automated  system  could  sig¬ 
nificantly  impact  the  classification  ability  of  the  system.  A  complete  set  of  designed 
experiments  should  be  conducted  with  the  AFPPM  to  determine  what  factors  sig¬ 
nificantly  influence  its  performance  and  how  to  manipulate  the  variables  to  achieve 
optimum  processing  for  automated  inspection. 

4.  Additional  pertinent  classification  features  for  identifying  cracked  blades  could  be 
extracted  from  the  bleed  out  characteristics  of  fluourescent  penetrant.  Research 
could  continue  to  better  determine  how  penetrant  bleeds  out  of  cracks  over  time. 
This  type  of  investigation  may  allow  for  a  more  accurate  characterization  of  the  FP 
process  and  lead  to  new  features  which  could  be  used  to  identify  cracks, 

5.  The  light  source  used  to  excite  the  penetrant  affects  not  only  the  way  the  penetrant 
reacts,  but  also  the  sensor's  ability  to  capture  the  fluorescent  image.  Another  research 
avenue  should  be  identifying  an  optimum  light  source  for  the  CCD  sensor  that  will 
improve  an  automated  system’s  ability  to  detect  crack  indications.  The  selection  of  a 
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light  source  must  consider  the  output  power  and  wavelength,  the  camera's  sensitivity, 
and  the  filters  available  to  improve  the  quality  of  the  light  incident  on  the  blade 
surface. 

6.  Removing  the  requirement  to  use  fluourescent  penetrant  preprocessing  could  poten¬ 
tially  reduce  the  time  and  money  spent  on  blade  inspection.  Research  should  at¬ 
tempt  to  identify  alternative  techniques  that  do  not  require  fluorescent  penetrant  to 
inspect  compressor  and  turbine  blades  in  a  production  environment.  SA-ALC  should 
continue  monitoring  progress  in  the  non-destructive  inspection  field  and  search  for 
developments  that  may  improve  their  capability. 

'In  its  present  form,  the  Integrated  Blade  Inspection  System  has  room  for  significant 
improvements.  The  above  recommendations  outline  the  research  steps  required  to  develop 
an  automated  inspection  system  that  has  the  potential  to  improve  the  overall  jet  engine 
blade  inspection  process  at  SA-ALC. 
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Appendix  A.  Interaction  Diagrams 


The  following  diagrams  illustrate  the  relationship  between  the  various  functional 
components  of  the  Integrated  Blade  Inspection  System  (IBIS).  The  first  portion  of  each 
figure  shows  the  overall  relationship  and  has  a  single  block  outlined.  The  second  half  of 
each  figure  displays  an  expanded  version  of  the  outlined  block. 

Each  detailed  diagram  shows  two  types  of  inputs  and  outputs:  functional  and  design. 
The  functional  inputs  and  outputs  are  the  products  the  unit  processes  when  it  is  operating 
as  part  of  the  IBIS.  For  example,  the  image  processing  and  classification  software  receives 
blade  data  from  the  sensor  and  determines  the  blade  disposition  as  an  output.  The  design 
inputs  are  what  must  be  known  in  order  to  develop  the  subsystem.  The  inputs  can  originate 
from  specifications,  technical  orders,  desires  of  SA-ALC,  or  from  other  design  blocks.  The 
design  outputs  are  what  is  known  about  the  system  after  that  unit  is  developed.  The 
outputs  may  be  related  to  other  design  blocks  and  serve  as  inputs  or  they  could  be  used 
to  determine  system  performance.  Note,  for  simplicity,  the  first  portion  of  each  figure  only 
shows  the  functional  relationships  between  the  blocks. 

These  diagrams  do  not  replace  the  interaction  matrices  developed  during  the  systems 
engineering  study,  but  they  do  serve  as  an  additional  tool  to  help  understand  the  design 
process  for  the  blade  inspection  system. 
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Figure  A.l.  AFPPM  subsystem. 
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Figure  A.2.  StatisticaJ  quality  control  process. 
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Functional  Flow  Diagram 
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Figure  A. 3.  Manipulator  subsystem. 
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Funolional  Flow  Diagmm 
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Figure  A,4.  Sensor  subsystem. 
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Functional  Flow  Diagram 
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Figure  A.5.  Image  processing  and  classification  software. 
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Functional  Flow  Diagram 
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Figure  A. 6.  Computer  subsystem. 


Appendix  B.  Statistical  Quality  Control 


B.  1  Overview 

A  commitment  to  a  coherent  Statistical  Quality  Control  (SQC)  plan  is  the  primary 
recommendation  under  alternative  1.  The  primary  end  of  this  SQC  presentation  is  to 
initiate  a  complete  analysis  of  the  quality  of  the  IBIS  process.  The  two  questions  that  must 
be  asked  are,  does  management  desire  to  instill  an  atmosphere  of  continual  improvement, 
and  should  the  IBIS  represent  the  model  jet  engine  blade  inspection  system  for  the  Air 
Force?  If  the  answers  to  these  questions  are  in  the  affirmative,  then  the  IBIS  process  needs 
a  comprehensive,  coherent  SQC  plan. 

The  current  system  operates  in  an  uncontrollable  manner.  The  critical  variables  are 
neither  adequately  identified  nor  are  they  monitored.  The  state  of  control  for  the  IBIS 
process  is  unknown,  and  the  process  performance  is  only  estimated.  This  unknown  state 
of  the  process  causes  the  system  performance  to  be  unquantifiable.  This  deficiency  leads 
to  two  related  problems.  First,  if  the  system  operates  out  of  control  or  if  the  system  fails 
to  meet  its  performance  requirements,  the  NDI  engineers  can  not  identify  that  a  problem 
exists,  and  therefore,  they  are  unable  to  fix  it.  Second,  the  approach  toward  continuous 
process  improvements  is  not  undertaken  in  a  systematic  manner.  The  implementation  of 
a  comprehensive,  coherent  SQC  plan  can  help  to  eliminate  these  system  shortcomings. 

The  suggested  SQC  plan  consists  of  three  sections;  random  sampling,  calibration 
experiments,  and  the  utilization  of  design  of  experiment  methods.  The  random  sampling 
generates  a  better  understanding  of  the  IBIS  product,  inspected  jet  engine  blades.  The 
calibration  experiments  show  if  the  system  meets  performance  requirements.  The  design 
of  experiments  methodology  identifies  critical  factors  whose  variability  must  be  closely 
controlled.  Together,  these  three  sections  of  the  SQC  plan  help  the  engineers  maintain 
process  control,  insure  acceptable  system  performance,  identify  problems,  and  implement 
system  improvements. 

The  entire  SQC  concept  revolves  around  two  critical  issues:  system  performance 
and  process  control  (17).  The  system  performance  must  insure  a  product  that  meets 
stipulated  design  requirements.  For  the  IBIS,  these  requirements  must  be  established  by 
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the  jet  engine  design  engineers  in  consultation  with  the  NDI  engineers.  Jet  engine  safety 
must  be  insured  from  the  design  perspective,  and  the  inspection  system  must  be  capable 
of  attaining  the  established  goals.  The  following  variables  represent  the  IBIS  process 
performance  measures:  the  probability  of  detection,  missed  detection  rates,  false  call  rates, 
product  quality,  and  scrap  quality.  Process  control  must  insure  the  system  output  variation 
falls  within  acceptable  parameters.  The  system  operation  is  out  of  control  if  the  fluctuation 
of  the  system  output  over  time  is  large  or  if  the  system  output  variation  is  dependent  on 
time.  The  lack  of  process  control  can  occur  regardless  of  the  average  output  performance 
level.  The  IBIS  must  insure  both  process  control  and  acceptable  performance  if  it  is  to 
represent  a  quality  system. 

Even  if  the  system  operates  in  a  state  of  control  and  produces  an  acceptable  product, 
the  quality  control  job  is  still  not  done.  Continual  improvements  to  the  system  represent 
a  cosisiderable  portion  of  the  SQC  job.  Improvements  must  be  diligently  sought,  proven, 
implemented,  and  verified.  The  SQC  plan  documents  process  control  and  system  perfor¬ 
mance  over  time.  The  SQC  plan  also  calls  for  knowledge  of  the  critical  variable  states 
over  time.  Superimposing  the  process  control  and  system  performance  information  with 
the  critical  variable  states  will  offer  information  on  trends  or  system  problems.  This  in¬ 
formation  will  immeasurably  aid  the  experienced  and  creative  NDI  engineer  to  identify 
problems,  implement  solutions,  and  improve  the  system. 

B.2  Performance  Measures 

B.2.1  Introduction.  The  probability  of  detection  (POD)  analysis,  miss  detection 
rate,  false  call  rate,  product  quality,  scrap  quality,  and  acceptable  quality  level  represent 
the  IBIS  process  performance  measures  for  the  IBIS  Upgrade  Study  SQC  plan.  Each 
measure  will  be  outlined  before  addressing  the  SQC  plan  in  more  detail. 

B.2. 2  Probability  of  Detection.  The  critical  performance  requirement  for  the  IBIS 
is  established  by  the  POD  analysis  as  defined  by  Dr  A1  Berens  (4). 


B-2 


The  POD{a)  function  is  defined  as  the  proportion  of  all  cracks  of  size  a 
that  will  be  detected  in  a  particular  application  of  an  NDE  system.  Assume 
that  each  crack  of  size  a  in  the  potential  population  of  cracks  has  its  own  dis¬ 
tinct  crack  detection  probability,  p,  and  that  the  probability  density  function  of 
the  detection  probabilities  are  given  by  /a(p).  [Figure  B.l]  shows  a  schematic 
representation  of  this  density.  The  conditional  probability  of  a  randomly  se¬ 
lected  crack  from  the  population  having  detection  probability  of  p  and  being 
detected  at  the  inspection  is  given  by  pf^{p)dp.  The  unconditional  probability 
of  a  randomly  selected  crack  from  the  population  being  detected  is  the  sum  of 
the  conditional  probabilities  over  the  range  of  p,  that  is: 

POD{a)=  {  pfa{p)dp  (B.l) 

Jo 

therefore,  POD{a)  is  the  average  of  the  detection  probabilities  for  cracks  of 
size  a. 


Figure  B.l.  POD  curve 


The  POD(a)  function  calculates  the  average  probability  of  detecting  a  crack  of  a 
given  length  a.  The  POD  analysis  then  plots  this  average  POD  valu^  for  many  different 
crack  sizes.  This  plot  represents  the  average  POD  curve  for  the  system  at  the  time  of  the 
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experiment.  The  average  POD  curve  is  also  referred  to  as  the  50  POD  curve.  The  POD 
analysis  also  generates  a  95%  lower  confidence  bound  on  this  average  POD  curve.  The 
lower  confidence  bound  curve  describes  the  value  where  there  is  a  95%  probability  that  the 
average  POD  curve  is  larger.  The  95%  lower  confidence  bound  for  the  POD  curve  is  also 
referred  to  as  the  95  POD  curve. 

The  engine  requirements(  15)  stipulate  that  the  human  blade  inspection  process  will 
detect  a  .070  inch  crack  with  a  90%  POD  at  the  95%  confidence  level.  This  requirement  is 
also  stated  as  the  inspection  system  must  detect  a  .070  inch  crack  with  a  90/95  POD  for 
manual  inspection. 

B.2.3  Missed  Detection  and  False  Call  Rates.  The  SQC  plan  calls  for  two  other 
related  performance  measures:  type  1  and  type  2  errors.  Under  the  assumption  that  each 
blade  is  cracked,  the  system  must  prove  that  the  blade  is  not  cracked  before  a  blade  can  be 
placed  into  service.  This  assumption  stipulates  that  the  null  hypothesis  {Ho)  becomes:  the 
blade  under  inspection  is  cracked.  The  alternate  hypothesis  [Ha]  correspondingly  becomes: 
the  blade  under  inspection  is  free  of  cracks.  The  type  1  error  for  NDI  inspection  is  the 
classification  of  a  cracked  blade  as  acceptable.  The  type  2  error  for  NDI  inspection  is  the 
classification  of  an  acceptable  blade  as  cracked.  Figure  B.2  illustrates  this  hypothesis  test 
approach  to  blade  inspection. 

In  order  to  clarify  the  meaning  of  the  type  1  and  type  2  errors,  the  definitions  of 
missed  detection  and  false  call  emerge.  The  system’s  missed  detection  rate  is  the  prob¬ 
ability  of  accepting  a  blade  with  a  crack,  which  equates  to  the  type  1  error  previously 
described.  The  system’s  false  call  rate  is  the  probability  of  rejecting  an  acceptable  blade, 
which  equates  to  the  type  2  error  previously  described. 

An  understanding  of  how  the  actual  false  call  rate  and  missed  detection  rate  manifest 
themselves  becomes  important.  The  variable  a(a)  describes  the  missed  detection  rate  of 
the  system.  The  0(0)  value  is  the  probability  of  missing  a  crack  of  length  a  or  greater.  The 
variable  /3(a)  describes  the  false  call  rate  of  the  system.  The  /3(a)  value  is  the  probability  of 
rejecting  a  crack  of  length  less  than  a.  Through  the  use  of  the  Q(a)  and  the  3{a)  variables. 
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Figure  B.2.  Hypothesis  test  errors 

the  NDI  engineer  can  examine  the  missed  detection  and  false  caJl  implications  at  various 
crack  lengths. 

In  order  to  establish  performance  reqmrements  for  missed  detection  and  false  call 
rates,  the  analysis  must  first  relate  the  known  POD(a)  requirements  to  a(o)  values.  The 
engine  specification  for  the  FIDO  engine  (15)  requires  manual  blade  inspection  processes 
to  operate  at  a  90/95  POD  for  .070  inch  cracks.  Therefore,  it  is  valid  to  define  the  missed 
detection  performance  requirement  as  a(.070)  <  .10  if  the  POD(a)  curve  is  monotonically 
nondecreasing. 

Since  the  IBIS  process  is  more  sensitive  to  missed  detections  than  false  calls  (false 
calls  are  much  less  expensive  to  the  Air  Force  than  missed  detections  (see  appendix  D)  the 
desired  /3(a)  value  will  evolve  as  the  system  becomes  characterized. 

The  a{a),  /3(a),  and  POD(a)  values  are  related.  If  the  POD(a)  value  increases,  then 
the  miss  detection  rate  will  decrease.  If  the  missed  detection  rate  decrecises,  the  false 
call  rate  typically  increases.  This  relationship  depends  on  the  process  performance.  If 
the  system’s  critical  variables  are  changed  such  that  POD  increases,  missed  detections 
decrease,  and  false  calls  decrease,  then  the  system  has  been  improved.  This  situation 
represents  an  improvement  to  the  IBIS  process.  If  the  improvement  can  be  realized  and 
the  process  controlled,  a  successful  step  towards  system  improvement  results. 
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B.2.4  Product  Quality,  Scrap  Quality,  and  Acceptable  Quality  Level.  Two  impor¬ 
tant  performance  measures  that  directly  result  from  the  random  sampling  inspection  are 
the  product  quality  and  the  scrap  quality.  The  product  quality  represents  the  percentage 
of  blades  the  IBIS  process  identifies  as  acceptable  that  are  actually  cracked.  The  scrap 
quality  represents  the  percentage  of  blades  the  IBIS  process  identifies  as  rejectable  that 
are  actually  cracked.  The  SQC  plan  calls  for  the  maximization  of  scrap  quality  value,  and 
the  minimization  of  the  product  quality  value.  The  random  sampling  inspection  directly 
determines  these  performance  measures  by  sampling  from  the  accepted  blades  and  rejected 
blades,  and  by  estimating  the  percentage  cracked  in  the  two  categories.  Since  the  product 
quality  and  scrap  quality  are  directly  determined,  they  become  the  natural  candidates  to 
measure  process  control.  These  values  also  contain  valuable  information  regarding  process 
performance.  Due  to  the  conclusion  that  product  quality  and  scrap  quality  offer  infor¬ 
mation  pertaining  to  both  process  control  and  process  performance,  they  will  be  plotted 
over  time  using  the  Shewhart  p-chart  (17,  20).  This  analysis  will  offer  a  temporal  view  of 
process  performance.  Figure  B.3  illustrates  an  example  p-chart. 


SampiM  In  Ontor  ol  Production 
(Random  aamplaa  twice  oadi  woak] 


Figure  B.3.  Example  p-chart. [Duncan:1974] 

The  Acceptable  Quality  Level  (AQL)  performance  measure  establishes  the  retjuire- 
ment  on  the  IBIS  process  to  produce  an  outgoing  product  that,  over  time,  will  have  no 
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greater  than  the  acceptable  percentage  of  cracked  blades.  The  NDl  engineers  must  trans¬ 
late  the  design  engineers  inspection  requirements  into  an  AQL  for  the  IBIS  process.  The 
IBIS  must  possess  the  capability  to  consistently  meet  this  requirement  or  else  jet  engines 
will  be  placed  at  an  unacceptable  risk  of  either  an  unscheduled  maintenance  cycle  or  a 
catastrophic  failure. 


B.2.5  Calculation  of  a{a)  and  (3{a).  The  random  sampling  procedure  provides  in¬ 
formation  regarding  the  proportion  of  defective  blades  that  the  process  has  identified  as  ac¬ 
ceptable  and  the  proportion  of  defective  blades  that  the  process  has  identified  as  rejectable. 
The  product  quality  is  determined  by  estimating  the  percentage  of  blades  accepted  by  the 
IBIS  process  that  are  actually  defective,  and  the  scrap  quality  is  determined  by  estimating 
the  percentage  of  blades  rejected  by  the  IBIS  process  that  are  actually  defective.  The 
product  quality  and  the  scrap  quality  are  conditional  probabilities.  The  product  quality  is 
the  probability  a  blade  is  cracked  given  the  system  has  accepted  it  { P{cracked\accepted)), 
and  the  scrap  quality  is  the  probability  a  blade  is  cracked  given  the  system  has  rejected  it 
(P(crack€d\rejecied)).  These  values  are  important  in  themselves,  but  they  do  not  repre¬ 
sent  the  missed  detection  rate  or  the  false  call  rate. 

The  missed  detection  and  false  call  rates  are  also  conditional  probabilities.  The 
missed  detection  rate  is  the  probability  the  system  will  accept  a  blade  given  it  is  cracked 
{P{accept\crack€d)).  The  false  call  rate  is  the  probability  the  system  will  reject  a  blade 
given  it  is  not  cracked  {P{r€ject\not  cracked)).  Bayes  rule  (see  equation  B.2)  provides  the 
means  of  calculating  the  missed  detection  rate  and  the  false  call  rate  from  the  outgoing 
quality  and  the  scrap  quality. 


P{A\B)  = 


PiB\A)  X  P(A) 

{P{B\A)  X  P{A)  d-  P{B\A)  X  P{A)) 


(B.2) 


The  following  example  calculations  illustrate  the  use  of  Bayes  rule  for  the  anal¬ 
ysis  of  random  sample  inspection  for  the  IBIS  process.  The  framework  of  the  exam¬ 
ple  requires  the  following  assumptions.  First,  assume  that  the  IBIS  process  has  gen¬ 
erally  identified  5%  of  the  blades  inspected  as  cracked.  The  results  of  the  hypotheti¬ 
cal  random  sampling  inspection  has  demonstrated  that  for  all  the  blades  accepted  by 
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the  IBIS  process  2%  actually  have  cracks  of  length  greater  than  .070  inches,  and  that 
for  all  the  blades  rejected  by  the  IBIS  process  97.5%  actually  have  cracks  of  length 
greater  than  .070  inches.  These  results  yield:  P{Tejected)  =  .05,  P(accepted)  —  .95, 
P{cracked\r ejected)  =  .975,  P(notcTacked\rejecied)  =  .025,  P{cracked\accepted)  =  .02. 
P{notcracked\accepted)  =  .98.  Now  applying  Bayes  rule,  the  following  calculations  result: 


P(accept\cracked)  = 


P{cracked\accepted)  x  P{accepted) 


P{cracked\accepied)  x  P{accepted)  +  P{cracked\r ejected)  X  P{rejected) 

(.02  X  .95) 


(B.3) 


a(.070)  = 


(.02x  .95)  +  (.975x  .05) 


=  .2804 


and 


P(reject\not  cracked)  = 

P(not  CTacked\rejected)  x  P^rejected) 

P(not  crackedjrejected)  x  P(rejected)  +  P(not  CTacked\accepted)  x  P(accepted) 


(B.4) 


yS(,070) 


.025  X  .05 

(.025  X  .05)  +  (.98  X  .95) 


=  .0013 


If  5%  of  the  blades  inspected  are  identified  as  cracked,  the  product  quality  is  determined 
to  be  2%,  and  the  scrap  quality  is  determined  to  be  97.5%  then  the  missed  detection  rate 
at  .070  inches  is  28%  and  the  false  call  rate  is  .1%.  The  POD(.070  inches)  is  expected  to 
be  less  than  72%. 


The  NDI  engineer  requires  both  forms  of  this  information.  The  scrap  quality  and 
the  product  quality  will  be  graphically  represented  using  the  Shewhart  chart  for  fraction 
defective  attribute  data  (the  p-chart).  This  tool  will  graphically  present  the  temporal 
performance  of  the  IBIS  process  (20).  The  control  charts  will  illustrate  if  the  process 
meets  its  performance  requirements  and  if  the  process  is  in  control. 

The  missed  detection  rate  and  the  false  call  rate  provide  valuable  insights  related  to 
the  POD(a)  analysis.  The  POD(a)  curve  illustrates  the  probability  the  process  will  find 
cracks  of  given  crack  lengths,  while  the  missed  detection  rate  provides  information  on  the 
probability  of  correctly  classifying  a  crack  of  a  given  length  or  greater.  The  false  call  rate 
identifies  the  probability  of  incorrectly  classifying  an  acceptable  blade  as  rejectable.  This 
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measure  controls  the  analysis  from  inflating  the  POD(a)  and  a(a)  values  by  rejecting  many 
acceptable  blades  along  with  the  cracked  blades. 

The  performance  measures  will  be  utilized  by  the  SQC  plan  to  quantify  the  IBIS 
process.  The  SQC  plan  will  attain  these  values  through  random  sampling  and  calibration 
experiments. 

B.3  Random  Sampling 

B.3.1  Introduction.  The  random  sampling  of  inspected  engine  blades  will  allow 
the  calculation  of  the  missed  detection  rate,  the  false  call  rate,  and  will  illustrate  the 
performance  over  time  of  the  IBIS  process.  The  random  sampling  portion  of  the  SQC  plan 
has  three  separate  inspection  schemes. 

The  first  inspection  scheme  calls  for  a  complete  sampling  plan  on  one  or  two  fracture 
critical  blade  types.  MIL-STD-105E  will  define  the  sample  plan  to  be  used  for  the  selected 
blade  types  (13).  The  standard’s  sampling  plans  require  the  establishment  of  inspection 
lots,  and  the  IBIS  process  must  protect  the  integrity  of  these  lots.  The  blades  identified  as 
acceptable  are  randomly  sampled  according  to  the  sampling  plan.  These  sampled  blades  are 
then  inspected,  and  the  analysis  of  the  inspection  result  will  determine  the  average  quality 
level.  The  inspection  also  provides  information  pertaining  to  the  process  quality  and  missed 
detection  rates.  This  information  provides  a  means  of  recording  the  system  performance. 
The  SQC  plan  therefore  identifies  when  the  system  performance  is  unacceptable  based 
on  sampling  from  blade  types  identified  as  critical.  Unacceptable  performance  equates  to 
system  failures.  Once  these  system  failures  emerge,  the  NDI  engineers  must  identify  the 
causes  and  fix  the  problem.  If  the  system  failures  are  not  readily  identified,  the  likelihood 
of  jet  engine  problems  related  to  turbine  and  compressor  blades  increase. 

A  large  assumption  in  this  first  random  sampling  scheme  lies  in  the  proposition  that 
the  few  blade  types  chosen  for  this  inspection  represent  all  the  blade  types  inspected.  This 
assumption  may  be  reasonable,  but  the  extrapolation  of  information  for  all  the  blades  from 
a  limited  portion  of  them  opens  the  danger  of  optimizing  the  system  for  a  characteristic 
particular  to  the  sampled  blade  types. 
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In  order  to  protect  against  this  danger,  the  SQC  plan  establishes  the  second  random 
sampling  scheme.  This  scheme  calls  for  the  random  sampling  of  all  outgoing  blades.  These 
sampled  blades  are  inspected,  and  the  percentage  of  cracked  blades  identified  by  the  system 
as  acceptable  is  established.  This  information  relates  to  the  product  quality  and  missed 
detection  rate  performance  measures.  These  values  must  fall  within  acceptable  parameters 
along  with  the  information  derived  from  the  first  random  sampling  scheme. 

The  third  random  sampling  scheme  calls  for  the  inspection  of  a  sizable  portion  of  the 
blades  identified  as  unacceptable.  This  scheme  serves  a  dual  purpose.  First,  it  determines 
the  percentage  of  blades  that  are  actually  cracked  out  of  all  the  blades  rejected  by  the 
process.  This  gives  information  pertaining  to  the  scrap  quality  and  false  call  rate  perfor¬ 
mance  measures.  These  values  must  fall  within  acceptable  limits  or  the  SQC  plan  will  call 
for  corrective  actions.  Secondly,  the  blades  truly  identified  as  cracked  can  be  characterized 
and  used  as  a  reference  standard  in  the  calibration  experiments,  the  second  part  of  the 
SQC  plan. 

B.3.2  Method  1:  Comprehensive  Sampling  Plan  of  Fracture  Critical  Blade  Stages. 

B. 3.2.1  Introduction.  This  method  of  drawing  statistical  information  perti¬ 
nent  to  process  performance  is  based  on  performing  a  comprehensive  statistical  analysis  of 
outgoing  parts  for  one  or  two  fracture  critical  blade  stages.  The  inspection  system  oper¬ 
ating  engineers  must  implement  a  sampling  plan  from  MIL-STD-105E;  they  must  utilize 
an  inspection  technique  fully  capable  of  identifying  any  cracks  present  in  the  sample  set; 
they  must  insure  acceptable  quality  levels  for  the  final  product.  The  inspection  results 
must  also  be  utilized  to  measure  the  product  quality  and  demonstrate  acceptable  process 
control.  This  information  will  also  be  important  in  the  calculation  of  the  missed  detection 
rates.  The  information  gained  must  then  serve  as  a  feedback  mechanism  to  identify  process 
deficiencies  and  to  identify  potential  areas  for  improvements. 

B.3.2. 2  Sampling  Plan.  Mil-Std-105E  establishes  lot  or  batch  sampling  plans 
and  procedures  for  inspection  by  attributes.  The  standard  presents  its  applicability  as 
follows  (13): 
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Sampling  plans  designated  in  this  publication  are  applicable,  but  not  limited 
to  inspection  of  the  following:  a.  end  items,  b.  components  and  raw  material, 
c.  operations  or  services,  d.  materials  in  process,  e.  supplies  in  storage,  f. 
maintenance  operations,  g.  data  or  records,  h.  administrative  procedures. 

The  use  of  this  standard  is  discussed  by  Archeson  Duncan  (20).  He  presents  seven 
steps  to  implement  a  sampling  plan  using  MIL-STD-105. 

1.  decide  on  an  AQL 

2.  decide  on  an  inspection  level 

3.  determine  lot  size 

4.  enter  table  to  find  sample  size  code  letter 

5.  decide  on  type  of  sampling  plan  to  be  used 

6.  enter  proper  table  to  find  the  plan  to  be  used 

7.  enter  a  tightened  inspection  for  the  same  type  of  plan  whenever  a  shift  to  tightened 
inspection  is  called  for 

The  first  sampling  method  will  follow  these  steps  to  present  a  potential  sample  plan 
for  the  IBIS. 

First,  the  acceptable  quality  level  must  be  chosen  by  design  and  inspection  engineers. 
The  design  documents  for  the  FlOO  engine  (15)  stipulate  a  .070  inch  crack  must  be  detected 
with  a  90%  POD  at  a  95%  confidence  level.  This  requirement  does  not  directly  address 
AQL  since  a  bad  part  is  defined  as  a  part  with  a  crack  greater  than  or  equal  to  .070  inches. 
The  POD  requirement  ensures  with  a  95%  confidence  that  .070  inch  cracks  can  be  detected, 
on  the  average,  90%  of  the  time.  Larger  cracks  can  be  detected  with  a  higher  POD  if  the 
process  POD  curve  is  monotonically  nondecreasing.  Therefore,  the  missed  detection  rate 
is  less  than  10%.  Realizing  that  the  missed  detection  rate  is  a  conditional  probability  as 
described  in  section  B.2.5,  the  product  quality  calculations  result  in  a  value  less  than  1%. 
Selecting  the  AQL  as  1%  offers  the  sampling  plan  a  good  initial  value. 

Once  the  AQL  has  been  selected,  an  inspection  level  must  be  chosen.  The  inspection 
level  determines  the  sample  size,  and  therefore,  determines  the  statistical  significance  of 
the  calculations.  Mil-Std-105E  states,  “The  inspection  level  determines  the  relationship 
between  the  lot  or  batch  size  and  the  sample  size.”  The  document  advises  the  implemen¬ 
tation  of  inspection  level  2  as  appropriate  for  most  cases.  For  systems  that  require  greater 
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discrimination  the  standard  calls  for  level  3  inspection.  For  systems  that  can  tolerate  less 
discrimination  the  standard  allows  the  implementation  of  inspection  level  1.  Level  1  in¬ 
spection  is  appropriate  for  the  IBIS  process  due  to  the  cost  associated  with  the  random 
sample  inspection,  and  the  fact  that  inspection  of  the  outgoing  parts  for  the  IBIS  has  not 
been  identified  as  critical  by  IBIS  management.  If  the  results  of  this  random  sampling 
scheme  prove  worthwhile,  a  switch  to  inspection  level  2  should  be  considered. 

The  next  step  calls  for  the  determination  of  lot  size.  This  method  will  design  around 
the  ninth  stage  compressor  blade.  According  to  the  available  data,  the  IBIS  processed 
11,310  ninth  stage  compressor  blades  in  1991.  Assuming  this  is  representative  of  yearly 
rates,  these  blades  could  be  broken  down  into  approximately  100  lots  of  between  90  to  150 
blades.  It  is  important  to  realize  that  the  ninth  stage  blades  will  not  be  passed  through  the 
AFPPM  until  there  are  between  91  and  150  blades  available.  Mil-Std-105E  requires  that 
the  lots  must  not  be  broken  up  if  the  resultant  information  is  to  be  statistically  correct.  It 
would  be  better  to  reduce  the  lot  size  than  to  split  up  a  lot. 

The  fourth  step  calls  for  the  determination  of  an  appropriate  sample  code  letter.  The 
lot  size  of  91-150  calls  for  a  code  letter  of  D  in  inspection  level  1. 

The  next  step  requires  a  decision  on  the  type  of  sampling  plan  to  be  used.  The 
standard  allows  single,  double,  and  multiple  sampling  plans.  The  single  sampling  plan 
calls  for  the  inspection  of  one  sample  set  of  a  determined  size.  If  the  number  of  defects  is 
acceptable,  the  lot  is  accepted.  The  double  sampling  plan  calls  for  a  half  size  sample  set  to 
be  drawn  from  the  lot  and  inspected.  If  the  number  of  defects  is  equal  to  or  less  than  the 
acceptance  number,  the  lot  is  accepted.  If  the  number  of  defects  is  between  the  acceptance 
number  and  the  rejection  number,  a  second  sample  is  drawn.  If  the  number  of  defects  is 
greater  than  the  rejection  number,  the  lot  is  rejected.  If  a  second  sample  was  required, 
the  total  number  of  defects  from  the  first  added  to  the  defects  from  the  second  sample 
set  must  be  below  the  given  rejection  number  for  the  lot  to  be  accepted.  The  multiple 
inspection  plan  is  similar  except  more  than  two  sample  sets  are  possible.  This  method  will 
use  a  single  sampling  plan  for  the  inspection  of  ninth  stage  compressor  blades  (13). 
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The  sixth  step  calls  for  the  selection  of  the  appropriate  table  and  the  determination 
of  the  appropriate  inspection  plan.  Utilizing  table  II- A,  Single  Sampling  Plans  For  Normal 
Inspection  (Master  Table)  (13),  the  sample  plan  dictated  by  code  letter  D  and  AQL  of 
1%  has  a  sample  size  of  50.  This  sampling  plan  accepts  the  lot  if  there  are  0  or  1  defects 
present  in  the  inspection.  Lot  rejection  occurs  if  there  are  two  or  more  defects  found  in 
the  sample. 

A  reexamination  of  the  selection  of  the  lot  size  is  called  for.  For  the  case  described 
above,  it  is  feasible  that  the  sampling  plan  will  call  for  the  inspection  of  50  out  of  91 
blades  if  a  1%  AQL  is  desired.  Randomly  sampling  more  than  half  of  the  lot  is  obviously 
unacceptable.  The  1%  AQL  would  result  under  the  proposed  implementation  of  Mil-Std- 
105E  sampling  for  lots  as  large  as  3200  units.  Therefore,  the  lots  should  be  increased  to  an 
appropriate  level  so  the  inspection  requirement  is  decreased,  and  the  process  throughput 
requirements  are  maintained.  If  the  number  of  ninth  stage  blades  passing  through  the 
system  is  too  small  to  make  Mil-Std-105E  inspection  feasible  then  multiple  blade  types 
could  be  used  to  create  the  lots. 

The  seventh  step  calls  for  a  transition  to  tightened  inspection  if  the  results  dictate 
such  a  switch.  The  switching  conditions  are  stipulated  on  page  6  of  Mil-Std-105E. 

B.3.2.3  Performance  Measure  Derivation.  The  comprehensive  sampling  of 
the  critical  blade  types  in  accordance  with  Mil-Std-105E  will  not  directly  generate  an 
AQL  value  for  the  IBIS  process.  The  design  requirements  stipulate  the  required  IBIS 
process  AQL,  and  the  sampling  scheme  defined  by  MiI-Std-105E  will  verify  if  the  process 
actually  operates  at  the  required  .4QL.  If  the  process  fails  to  operate  at  the  stipulated 
AQL,  then  pertinent  changes  to  the  IBIS  process  must  be  implemented.  If  changes  to  the 
IBIS  are  not  feasible,  then  changes  to  the  stipulated  AQL  value  must  be  defended.  The 
IBIS  Upgrade  Study  recommends  the  implementation  of  improvements  to  the  system  over 
changing  stipulated  performance  measures. 

The  verification  of  an  AQL  level  through  the  use  of  Mil-Std-105E  comes  with  some 
statistical  risk.  In  other  words,  Mil-Std-105E  does  not  verify  with  certainty  that,  on 
average,  the  AQL  of  the  process’  product  will  be  less  than  the  stipulated  value.  The  level 
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of  this  risk  primarily  results  from  the  sample  size  in  question.  Inspection  level  1  has  more 
risk  than  level  2,  and  inspection  level  2  has  more  risk  than  level  3.  Mil-Std-105E  presents 
operating  characteristics  curves  that  illustrate  the  risk  associated  with  the  sampling  plans 
chosen. 

The  comprehensive  sampling  of  critical  blade  types  will  directly  generate  a  measure 
of  product  quality.  The  product  quality  derived  from  this  random  sample  inspection  will 
be  for  the  critical  blade  type  or  types  in  question.  The  sample  size  will  be  known  as  will 
the  number  of  blades  that  have  a  crack  of  length  greater  than  a.  The  product  quality  at 
a  is  calculated  using  equation  B.5  if  the  sample  is  taken  from  the  blades  identified  by  the 
IBIS  process  as  acceptable. 


PQ{a) 


rejectable  blades 
inspected  blades 


(B.5) 


Both  the  PQ{a)  value  and  the  time  the  sampling  occurred  are  recorded.  Each  PQ{a )  va'ue 
is  plotted  with  respect  to  time  using  the  Shewhart  control  chart  appropriate  for  fraction 
defective  attribute  data,  the  p-chart  (17,  20). 

The  p-chart  will  graphically  represent  fluctuations  in  the  product  quality  measure. 
The  purpose  of  the  p-chart  is  to  monitor  process  control  and  process  performance.  The 
process  performance  is  inferred  directly  from  the  PQ{a)  values  generated  by  the  random 
sampling  inspection.  The  p-chart  monitors  process  control  by  looking  for  two  situations 
that  an  out-of-control  process  will  demonstrate.  First,  the  process  fluctuations  should 
vary  about  a  nominal  value.  This  variability  must  be  statistically  acceptable.  It  would  be 
unlikely  that  any  PQ{a)  values  would  fall  three  standard  deviations  away  from  the  nominal 
value  for  a  process  operating  in  control.  If  a  PQ{a)  value  emerges  outside  of  three  standard 
deviations  from  the  nominal  value,  then  the  process  is  deemed  out-of-control.  Second,  the 
variability  of  the  process  should  be  entirely  random.  The  IBIS  process  will  be  deemed 
out-of-control  if  the  p-chart  shows  non-random  behavior.  Such  non-random  behavior  will 
manifest  itself  as  time  dependent  fluctuations  about  the  nominal  value.  Consistent  runs 
of  PQ{a)  values  above  or  below  the  nominal  value  are  cause  for  concern,  as  are  consistent 
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runs  of  either  increasing  or  decreasing  PQ(a)  values.  Tables  to  determine  if  any  apparent 
runs  are  statistically  significant  are  available  (20). 


The  comprehensive  sampling  of  the  critical  blade  types  offers  information  pertinent  to 
the  calculation  of  the  missed  detection  rate  for  the  blades  in  question.  However,  the  missed 
detection  rate  calculation  also  requires  information  from  the  random  sample  inspection  of 
the  blades  identified  a.s  cracked  by  the  IBIS  process. 

B.3.3  Method  2:  Sampling  Plan  of  all  Outgoing  Blades. 


B.3.3.1  Introduction.  This  method  of  drawing  statistical  information  perti¬ 
nent  to  process  performance  is  based  on  performing  a  random  sample  inspection  of  all 
outgoing  engine  blades.  An  outgoing  engine  blade  is  a  blade  that  the  IBIS  process  has 
identified  as  acceptable  and  in  turn,  will  be  placed  back  in  service.  The  NDI  engineers  must 
randomly  sample  outgoing  blades  over  an  established  time  frame;  they  must  accurately  in¬ 
spect  the  blades  for  cracks;  they  must  draw  pertinent  information  from  the  inspection 


results  and  use  the  information  gained  to  correct  problems  and  implement  improvements. 


B.3.3. 2  Sampling  Plan.  A  series  of  steps  are  presented  in  order  to  gather  the 
information  pertinent  to  generating  the  p-chart  for  outgoing  blades  and  to  calculating  the 
missed  detection  and  false  call  rates  for  outgoing  blades.  The  first  step  is  to  define  a  time 
interval  over  which  the  random  sampling  will  occur  and  generate  an  accurate  count  of  all 
the  blades  processed  by  the  IBIS.  An  automated  blade  counter  should  be  designed  to  fulfill 
this  requirement  (see  section  B.3.5). 

Once  the  total  number  of  blades  processed  by  the  IBIS  in  the  stipulated  time  is  ac¬ 
curately  determined,  the  random  sampling  plan  calls  for  an  accurate  count  of  the  blades 
identified  as  rejectable  by  the  IBIS  process  over  that  same  time  period.  A  statistically 
significant  number  of  blades  must  then  be  randomly  selected  from  the  outgoing  blades, 
and  these  blades  must  be  carefully  inspected.  A  fluorescence  microscope  should  be  used 
for  the  inspection  (see  section  B.3.6).  The  blades  with  any  measurable  cracks  must  be 


examined  closely.  The  SQC  inspector  must  accurately  measure  the  length  of  all  the  iden¬ 
tified  cracks,  and  the  cracked  blades  must  be  labeled  and  stored.  Six  pieces  of  information 
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must  be  entered  into  a  database  for  each  blade  identified  as  cracked  by  the  random  sample 
inspection. 


1.  the  blade’s  original  IBIS  classification 

2.  the  crack  length  found 

3.  the  time  that  the  blade  was  sampled 

4.  the  size  of  the  random  sample 

5.  the  number  of  blades  inspected  by  the  IBIS  for  the  specified  time  period 

6.  the  number  of  blades  the  IBIS  process  identified  as  cracked 

B.3.3.3  Performance  Measure  Derivation.  The  random  sampling  of  outgoing 
blades  offers  a  direct  measure  of  the  product  quality.  This  value  is  estimated  for  the  time 
period  in  question  by  dividing  the  number  of  rejectable  parts  (#R)  by  the  size  of  the 
random  sample  (#RS).  Equation  B.6  illustrates  this  relationship. 

=  ^  (b.6) 

Recalling  that  a  rejectable  blade  is  defined  by  crack  size  a,  the  IBIS  process  will  demon¬ 
strate  different  PQ(a)  as  the  minimum  allowable  crack  size,  a.  changes.  The  NDI  engineers 
must  then  plot  the  PQ{a)  values  using  the  p-chart  and  watch  for  process  control  problems 
(see  section  B.3.2.3). 

The  information  gathered  from  the  random  sample  inspection  of  outgoing  blades 
provides  vital  data  for  the  calculation  of  the  system  missed  detection  rate.  However,  the 
calculation  of  the  missed  detection  requires  information  from  the  random  sample  inspection 
of  the  blades  identified  by  the  IBIS  process  as  rejectable. 

B.3.4  Method  3:  Sampling  Plan  of  Rejected  Blades. 

B. 3.4.1  Introduction.  The  final  method  of  drawing  statistical  information 
pertinent  to  process  performance  is  based  on  performing  a  random  sample  inspection  of 
the  blades  identified  as  rejectable  by  the  IBIS.  Two  cases  emerge  for  the  random  sampling 
of  rejected  parts.  The  first  case  is  the  inspection  of  the  blades  selected  for  inspection  in 
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accordance  with  Mil-Std-105E.  The  second  case  is  the  inspection  of  all  other  jet  engine 
blades.  The  NDI  engineers  must  randomly  sample  the  rejected  blades  for  the  two  cases: 
they  must  accurately  inspect  the  blades  for  cracks;  they  must  measure  the  size  of  the  cracks: 
they  must  draw  pertinent  information  from  the  inspection  results  and  use  the  information 
gained  to  correct  problems  and  implement  improvements. 

B.3.4.2  Sampling  Plan.  The  sampling  plan  for  the  blades  rejected  by  the 
IBIS  process  is  the  same  as  the  sampling  plan  of  all  outgoing  blades  except  that  there 
are  two  cases  of  rejected  blades.  The  two  cases  are,  those  blade  types  inspected  under 
method  1  using  Mil-Std-105E  and  those  blades  inspected  under  method  2  sampled  from 
all  outgoing  blades.  The  two  cases  are  treated  the  same  in  terms  of  collecting  the  data,  but 
the  respective  results  must  be  maintained  separately.  Each  case  offers  different  information 
to  the  NDI  engineers.  Case  1  provides  insight  to  the  IBIS  process  performance  because  it 
represents  a  comprehensive  examination  of  a  few  blade  types  that  represent  all  the  blades 
inspected  by  the  system.  Also,  the  use  of  inspection  lots  will  standardize  the  information 
and  provide  information  about  the  process  at  a  known  time.  Case  2  offers  additional 
insight  to  the  performance  of  the  IBIS  process  because  it  occurs  on  a  continuous  basis  and 
examines  the  process  over  all  the  blade  types  inspected. 

The  SQC  plan  requires  a  large  percentage  of  the  rejected  blades  to  be  randomly 
sampled  and  carefully  inspected.  The  reason  the  SQC  plan  requires  a  large  percentage  of 
these  blades  rests  in  the  desire  to  generate  a  reference  standard  set  that  can  be  used  to 
calibrate  the  performance  of  the  IBIS  process.  Therefore,  the  random  sample  inspection  of 
the  blades  rejected  by  the  IBIS  process  for  case  1  requires  a  large  percentage,  say  50%,  of 
these  rejected  blades  for  any  given  lot.  For  case  2,  a  large  percentage  of  the  blades  rejected 
over  the  stipulated  time  period  are  randomly  sampled. 

Once  the  blades  are  sampled  the  SQC  plan  requires  the  same  information  fpr  method 
3  as  for  method  2.  The  same  six  pieces  of  information  collected  under  sampling  method  2 
must  be  entered  into  a  database  for  each  blade  identified  as  cracked  by  the  third  method 
of  random  sample  inspection. 
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The  question  that  may  arise  is,  why  does  the  SQC  inspector  look  for  cracks  from  the 
blades  rejected  by  the  IBIS  process?  In  order  to  answer  this  question  one  must  recall  that 
the  SQC  plan  defines  a  blade  as  cracked  (or  rejectable)  if  it  has  a  crack  of  length  greater 
than  a.  If  the  NDI  engineers  desire  the  ability  to  examine  the  performance  of  the  IBIS 
process  for  multiple  crack  sizes  they  must  have  access  to  information  pertaining  to  system 
performance  for  all  crack  sizes. 

B.3.4.3  Performance  Measure  Derivation.  The  random  sample  inspection  of 
the  rejected  blades  directly  estimates  the  scrap  quality. 

^  number  of  blades  with  cracks  >  a 

number  of  blades  sampled 

This  value  is  plotted  on  the  p-chart  versus  time.  The  NDI  engineer  then  monitors  for 
process  control  (see  sec  B.3.2.3). 

Th missed  detection  rate  and  false  call  rate  calculation  are  possible  once  the  scrap 
quality  and  product  quality  are  determined.  Different  false  call  rates  and  missed  detection 
rates  are  calculated  for  the  blade  types  inspected  under  Mil-Std-105E  and  for  the  blades 
sampled  from  all  blades  processed  by  the  IBIS.  These  calculation  are  performed  using 
equations  B.3  and  B.4. 

B.3.5  Automated  Blade  Counter.  The  SQC  plan  requires  an  accurate  count  of  all 
the  blades  processed  by  the  IBIS,  which  can  be  provided  by  an  automated  blade  counter. 
A  CCD  sensor  and  a  computer  represent  the  technological  needs  of  the  blade  counter. 
The  blade  counter  could  be  either  fully  automated  or  semi-automated  and  could  be  used 
anywhere  within  the  IBIS  as  long  as  all  the  blades  processed  are  counted.  Two  likely 
locations  for  a  fully  automated  counter  would  be  within  the  penetrant  dwell  portion  of 
the  IBIS  or  after  the  blades  leave  the  developer  module.  Two  likely  locations  fpr  a  semi- 
automated  blade  counter  would  be  prior  to  loading  the  baskets  onto  the  AFPPM  over 
head  conveyor,  or  inside  the  inspection  booth,  prior  to  the  human  inspection  for  relevant 
FP  indications.  A  semi-automated  counter  that  would  be  used  prior  to  loading  the  IBIS 
basket  onto  the  AFPPM  conveyor  should  work  well  for  the  manual  inspection  alternative. 
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The  operators  would  either  bring  the  basket  to  the  sensor  or  bring  the  sensor  to  the  basket. 
They  would  then  take  a  picture,  and  the  computer  would  generate  a  count. 

A  concern  in  regard  to  the  blade  counter  is  to  match  the  count  with  a  particular 
blade  type.  An  accurate  count  of  all  the  blades  processed  is  vital  for  the  SQC  plan.  A 
count  of  each  blade  type  processed  by  the  IBIS  would  also  provide  valuable  information. 
In  a  semi- automated  system  the  operator  must  maintain  the  order  of  the  baskets  from  the 
blade  counter  onto  the  AFPPM  conveyor.  If  the  order  is  maintained,  the  blade  counter's 
task  of  matching  the  generated  count  with  a  particular  blade  type  becomes  trivial.  Since 
each  basket  contains  a  single  blade  type,  and  the  AFPPM  requires  the  blade  type  for  each 
basket  in  order  to  initiate  the  appropriate  recipe  for  the  FP  processing,  the  counter  can 
determine  what  basket  corresponds  to  each  count  through  a  query  to  the  AFPPM  control 
computer. 

B.3.6  Inspection  Techniques.  The  inspection  of  the  randomly  sampled  blades  must 
fulfill  two  fundamental  requirements.  First,  the  inspection  technique  used  must  find  cracks 
with  an  extremely  high  confidence.  This  is  important  if  the  SQC  plan  desires  to  determine 
the  system  missed  detection  rate  and  false  call  rate  for  variable  crack  sizes,  a(a)  and  /?(o). 
Second,  the  inspection  technique  must  accurately  measure  crack  lengths. 

The  inspection  technique  will  attain  the  high  rate  of  confidence  for  the  identification 
of  cracks  by  performing  extensive  cleaning  of  the  blades  and  by  detecting  the  fluores¬ 
cent  penetrant  indications  through  the  use  of  fluorescence  microscopes.  The  fluorescence 
microscope  will  also  accurately  measure  the  length  of  the  crack  under  examination. 

The  Nikon  MM-llU  measurescope  is  suitable  for  the  random  sample  inspection  task. 
The  MM-llU  reflected  light  microscope  focuses  violet  light  with  wavelength  of  380-425 
nanometers  on  the  blade  under  inspection  (330-380  nm  Ultraviolet  filters  are  also  available). 
The  microscope  has  magnification  levels  of  75X  -  750X.  A  demonstration  of.  the  MM- 
llU  vividly  illustrated  the  capability  of  the  fluorescence  microscope  to  find  and  measure 
cracks.  The  blades  were  manually  scanned  at  the  lower  magnifications  and  the  fluorescent 
indications  were  readily  apparent.  Differences  between  cracks  and  other  indications  were 
obvious  even  at  the  lower  magnifications.  Once  a  crack  was  located,  the  operator  placed 
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one  end  of  the  crack  in  the  field  of  view.  The  magnification  was  then  increased  and  the 
filter  was  removed  by  switching  a  lever.  The  operator  accurately  found  the  other  end  of 
the  crack  by  using  the  increased  magnification  and  the  white  light  source.  The  measuring 
device  was  then  initiated  and  the  ultraviolet  filter  was  reengaged.  The  operator  followed 
the  crack  to  its  opposite  end,  switched  back  to  the  white  light,  and  accurately  found  the 
edge  of  the  crack.  The  crack  length  was  then  read  from  the  digital  meter.  The  measurement 
accuracy  was  1  x  10”“*  inches.  This  accuracy  exceeds  the  requirements  necessary  for  the 
random  sampling  measurement. 

B.4  Calibration  Experiments 

B.4-1  Overview.  The  calibration  experiments  provide  a  repeatable  means  to  ad¬ 
just  the  IBIS  and  demonstrate  that  the  system  meets  performance  requirements.  These 
experiments  generate  the  system  POD(a)  curves.  They  also  offer  a  method  that  provides 
a  quick  check  on  system  performance.  Rummel  presents  two  standard  tools  as  a  means 
of  controlling  the  NDI  process:  qualification  standards  and  reference  standards  (52).  The 
qualification  standard  represents  the  means  to  perform  the  POD  analysis  to  establish  the 
system  performance.  The  IBIS  qualification  standard  is  a  set  of  blades  with  artificially 
grown  cracks.  The  length  of  these  cracks  are  accurately  measured.  The  reference  standard 
represents  a  means  of  frequently  gauging  the  performance  capabilities  of  the  system.  The 
reference  standard  must  emulate  the  smallest  crack  the  IBIS  must  consistently  detect.  If 
the  reference  standard  fails  to  react  positively  then  the  process  is  suspect.  If  it  reacts  pos¬ 
itively  then  problems  are  not  identified,  and  the  NDI  engineer  has  reason  to  assume  that 
the  system  continues  to  meet  performance  requirements.  A  schedule  must  be  established 
that  calls  for  many  reference  standard  experiments  between  periodic  qualification  standard 
experiments. 

B.4.2  Qualification  Standards.  The  qualification  standard  for  the  IBIS  must  be  a 
blade  set  that  possess  a  statistically  significant  set  of  cracks.  The  crack  lengths  must  range 
from  virtually  undetectable  (quite  small  <  .04  inches)  to  readily  detectable  (quite  large 
>  .2  inches),  and  the  lengths  must  evenly  span  the  range.  The  IBIS  must  periodically 
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process  these  blades  in  accordance  with  the  POD  analysis  requirements  as  stipulated  in 
the  POD  Generation  Software  Users  Manual  (5)  and  in  the  yet  to  be  published  MIL-STD 
for  USAF  NDE  System  Reliability  Assessment  (16).  The  NDI  engineer  must  intermix  the 
qualification  standard  with  at  least  2  -  3  times  more  crack-free  blades.  The  experiment 
results  improve  as  the  ratio  of  crack-free  blades  to  cracked  blades  increases  (5).  The 
experiment  must  be  run  several  times  and  the  hit/miss  data  must  be  recorded.  The  POD 
Curve  Generation  Software  (created  by  the  University  of  Dayton  Research  Institute,  and 
available  from  ASC/ENFSF,  Wright  Patterson  AFB,  Ohio)  must  then  process  this  data 
and  produce  the  system  POD  curves.  Once  available,  the  NDI  engineer  must  analyze  these 
POD  curves  and  characterize  the  system  performance.  If  the  analysis  demonstrates  the 
system  performance  exceeds  the  minimum  standards  established  by  the  jet  engine  design 
engineers,  the  NDI  engineers  can  deem  the  process  acceptable.  If  this  is  not  the  case, 
the  critical  variables  for  the  inspection  system  must  be  adjusted,  and  the  qualification 
experiment  must  be  rerun  until  the  analysis  proves  acceptable  performance. 

B.4-3  Reference  Standards.  The  reference  standard  is  a  tool  utilized  between  qual¬ 
ification  experiments  that  offers  some  evidence  that  the  system  performance  meets  the 
stipulated  requirements.  This  standard  must  be  representative  of  the  in-service  parts  in¬ 
spected  on  a  regular  basis.  It  also  must  stress  the  system  in  such  a  manner  that  the  NDI 
engineer  will  possess  confidence  that  the  system  will  identify  cracks  larger  than  the  maxi¬ 
mum  crack  the  system  has  been  calibrated  to  miss.  In  other  words,  the  reference  standard 
must  stress  the  lower  limit  of  the  inspection  system.  The  NDI  engineer  will  then  possess 
a  level  of  confidence  that  the  system  performance  is  acceptable. 

The  information  derived  from  the  reference  standards  is  incomplete  and  serves  solely 
as  a  means  to  instill  confidence  in  the  system  performance.  The  qualification  experiment 
must  be  periodically  run  in  order  to  truly  demonstrate  acceptable  performance.  Also,  the 
qualification  experiment  must  be  run  before  and  after  any  changes  are  made  to  the  system 
or  the  process. 

In-service  blades  that  have  been  previously  characterized  as  cracked  by  the  IBIS  pro¬ 
cess  should  be  used  as  a  reference  standard.  The  random  sampling  methods  will  generate 
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in-service  blades  with  characterized  cracks.  K  a  small  number  of  these  blades  were  placed 
in  a  pre-established  lot  of  acceptable  blades  and  then  processed  by  the  IBIS,  the  NDI  en¬ 
gineer  can  attain  information  pertinent  to  system  performance.  A  correct  classification  of 
the  cracked  blades  in  the  reference  standard  lot  bolsters  the  NDI  engineers’  confidence  that 
the  system  performance  is  acceptable.  Similarly,  an  incorrect  classification  of  the  reference 
standard  notifies  them  of  potential  problems. 

Two  important  requirements  must  be  adhered  to  for  the  success  of  the  in-service 
blade  reference  standard.  First,  the  reference  standard  lot  must  closely  resemble  a  normal 
basket  of  blades  under  inspection.  The  reference  standard  lot  must  arrive  as  all  blades 
normally  arrive;  the  reference  standard  lot  must  be  processed  by  the  IBIS  as  are  any 
typical  blade;  the  reference  standard  lot  must  possess  a  percentage  of  cracked  blades  to 
acceptable  blades  similar  to  any  basket  of  blades  normally  inspected  by  the  IBIS.  The 
reference  standard  must  truly  stress  the  capabilities  of  the  IBIS  process,  and  any  aspect  of 
the  reference  standard  that  sets  it  apart  as  unique  wiU  limit  its  capability  to  calibrate  the 
system.  Second,  the  reference  standard  lot  must  be  closely  tracked  so  the  NDI  engineers 
can  attain  the  required  information  and  so  no  cracked  blades  can  mix  with  the  outgoing 
blades. 

The  random  sampling  of  inspected  blades  and  the  calibrr,tion  experiments  measure 
the  performance  of  the  IBIS  process.  There  are  many  variables  within  the  IBIS  process 
that  influence  the  system  performance.  If  the  NDI  engineers  controlled  the  fluctuation 
of  these  variables,  they  would  control  the  performance  of  the  process.  By  implementing 
designed  experiments  the  NDI  engineers  can  identify  the  significant  variables  of  the  IBIS 
process. 

B.5  Design  Of  Experiments 

B.5.1  Overview.  The  application  of  design  of  experiment  methods  to  the  analysis 
of  the  IBIS  process  em'^rges  as  an  integral  part  of  the  SQC  plan.  Montgomery  defines  a 
designed  experiment  as  “a  test  or  series  of  tests  in  which  purposeful  changes  are  made  to  the 
input  variables  of  a  process  or  system  so  that  we  may  observe  and  identify  the  reasons  for 
changes  in  the  output  response”  (40).  Through  the  use  of  designed  experiments,  significant 
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variables  within  the  IBIS  process  will  be  identified.  Variations  in  these  significant  process 
variables  will  have  a  direct  impact  on  the  performance  of  the  IBIS  process.  The  use  of 
designed  experiments  will  not  only  identify  significant  process  variables,  but  will  establish 
acceptable  ranges  of  the  parameters  associated  with  these  variables.  Over  time,  monitoring 
these  parameters  can  serve  as  a  means  to  insure  process  control  instead  of  the  use  of 
Shewhart  control  charts.  (40) 

The  IBIS  process  lends  itself  to  the  use  of  design  of  experiment  concepts.  The  product 
of  the  process  is  inspected  jet  engine  blades  that  are  free  of  cracks.  The  performance  of 
the  IBIS  process  is  measurable  by  monitoring  the  process  POD(a),  the  missed  detection 
rate,  the  false  call  rate,  the  product  quality  and  the  scrap  quality.  There  are  multiple 
variables  within  the  IBIS  process  that  impact  the  system  performance.  Some  examples 
of  these  variables  are  emulsifier  dwell  time,  emulsifier  concentration,  inspector  experience, 
and  vibratory  cleaning  of  the  inspected  blades.  The  goal  of  the  design  of  experiment 
portion  of  the  SQC  plan  is  to  identify  the  significant  process  variables  and  to  understand 
the  impact  of  fluctuations  in  the  parameters  associated  with  these  variables. 

B.5.2  Design  of  Ezpei  imertt  Implementation.  Several  texts  discuss  the  mechanics 
of  designed  experiments  ( 16,  17,  25,  40).  Forgoing  a  similar  discussion,  a  methodology  issue 
pertaining  to  designed  experiments  is  presented  instead.  Designed  experiments  must  be 
implemented  with  a  focus  on  a  long  range  goal.  One  or  two  designed  experiments  will  offer 
information  for  the  NDI  engineer,  but  the  gradual  use  of  design  of  experiment  methods  will 
offer  much  richer  information.  The  danger  in  designing  one  or  two  experiments  to  identify 
the  significant  variables  is  that  the  experiments  will  be  too  broad  and  overreaching.  The 
NDI  engineers  will  not  readily  extract  all  the  information  desired  from  such  an  experiment. 
Instead,  the  design  of  experiments  methodology  must  possess  an  iterative  nature  (40).  The 
focus  of  each  experiment  will  be  sharp,  and  the  information  gained  will  help  control  the 
IBIS  process  and  the  design  of  future  experiments. 

SA-ALC/TIEO  performed  an  initial  designed  experiment  for  the  IBIS  Upgrade  Study. 
The  information  gained  represents  a  good  point  from  which  to  design  a  second  experiment. 
Based  on  the  results  of  the  initial  experiment  and  knowledge  gained  through  the  upgrade 
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study,  a  second  experiment  that  could  be  run  would  be  a  four  factor  two  level  full  factorial 
experiment.  The  four  factors  recommended  are: 


1.  use  of  the  vibratory 

2.  age  of  emulsifier  used 

3.  emulsifier  dwell  time 

4.  inspector  experience  level 

The  result  of  this  experiment  would  show  which  of  these  variables,  if  any,  are  signif¬ 
icant  to  the  IBIS  process  performance.  If  a  variable  is  significant,  the  NDI  engineers  must 
insure  it  is  properly  monitored. 

B.6  SQC  Plan  Frequencies 

The  frequencies  of  the  random  sampling,  calibration  experiments,  and  design  of  ex¬ 
periments  is  central  to  the  success  of  the  SQC  plan.  Information  must  be  extracted  from 
the  system  at  a  frequency  that  will  detect  problems  and  ensure  the  process  meets  its 
performance  specifications.  However,  the  frequencies  used  by  the  SQC  plan  must  not  be 
excessive  because  the  related  costs  will  be  prohibitive.  A  proposal  for  the  random  sampling, 
calibration  experiments,  and  design  of  experiments  follows. 

B.6.1  Random  Sampling.  The  random  sampling  portion  of  the  SQC  plan  has  three 
methods:  MIL-STD-105E  sampling  scheme,  random  sampling  of  outgoing  blades,  and 
random  sampling  of  scrap  blades.  This  section  addresses  each  portion. 

The  frequency  of  MIL-STD-105E  sampling  is  defined  by  the  creation  of  batches  or 
lots  of  critical  blade  types.  When  enough  critical  blades  queue  before  the  system  to  create 
a  lot,  the  system  processes  and  inspects  the  blades.  The  appropriate  random  sample  is 
drawn  and  inspected.  The  determination  of  an  inspection  frequency  for  MIL-STD-105E 
under  the  1.2  million  blades  per  year  throughput  constraint  requires  a  few  assumptions. 
First,  the  SQC  plan  uses  ninth  stage  compressor  blades  as  the  critical  blade  type  for  the 
FlOO-229  engine,  and  the  proportion  of  critical  blades  in  other  engines  is  similar  to  the 
ninth  stage  in  FlOO-229  engines.  Second,  the  1.2  million  blades  enter  the  system  evenly 
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distributed  throughout  the  year.  These  assumptions  yield  1.985  critical  blades  entering 
the  system  per  week  (see  equation  B.7). 
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where: 

74  =  number  of  stage  9  compressor  blades  in  FlOO-PW-229  engines 

860  =  number  of  compressor  and  turbine  blades  in  FlOO-PW-229  engines 

A  lot  size  of  900  to  1000  critical  blades  emerge  as  appropriate  if  1,985  critical  blades 
enter  the  system  per  week.  Therefore,  two  MIL-STD-105E  lots  will  be  processed  and 
inspected  by  the  IBIS  per  week.  Fifty  blades  will  be  drawn  from  each  lot  and  the  SQC 
inspector  will  inspect  these  blades. 

The  random  sampling  of  outgoing  blades  occurs  twice  per  week.  Two  time  windows 
are  established  during  a  week.  The  time  windows  for  a  full  5-day  week  are  start  of  business 
Monday  through  midday  Wednesday,  and  midday  Wednesday  through  close  of  business 
Friday.  The  SQC  plan  calls  for  a  total  of  50  blades  to  be  randomly  sampled  throughout 
each  of  the  two  weekly  time  windows.  The  SQC  inspector  wiU  inspect  these  blades  and 
determine  the  product  quality. 

The  random  sampling  of  the  scrap  blades  occurs  twice  per  week  in  a  similar  manner 
to  the  sampling  of  outgoing  blades.  The  same  two  time  windows  are  used  and  a  total  of 
33  blades  are  randomly  sampled  from  the  scrap  blades  throughout  the  two  weekly  time 
windows.  The  SQC  inspector  will  inspect  these  blades  and  determine  the  scrap  quality. 

B.  6.2  Calibration  Experiments.  The  calibration  experiments  are  the  reference  stan¬ 
dard  experiment  and  the  qualification  standard  experiment.  These  two  experiments  occur 
at  different  frequencies.  The  qualification  experiment  represents  a  comprehensive  examina¬ 
tion  of  the  system’s  performance.  The  SQC  plan  calls  for  quarterly  qualification  standard 
experiments.  The  reference  standard  experiments  represent  a  quick  confidence  check  for 
the  NDI  engineers.  The  reference  standard  experiments  should  occur  at  least  three  times 
per  week. 
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B.6.3  Design  of  Experiments.  The  designed  experiments  should  occur  as  required 
by  the  NDI  Engineers  so  they  can  identify  critical  variables  and  effectively  implement 
improvements  to  the  IBIS  process.  The  SQC  plan  does  not  establish  a  schedule  for  the 
designed  experiments,  however,  several  designed  experiments  should  be  run  as  early  as  pos¬ 
sible  to  characterize  the  system,  identify  the  most  critical  variables,  and  suggest  immediate 
improvements. 

B.  7  Training 

The  SQC  plan  calls  for  continual  training  of  all  humans  involved  in  the  process. 
Everyone  must  have  a  keen  understanding  of  the  importance  of  the  process;  they  must 
understand  the  overall  process;  they  must  have  in-depth  knowledge  of  their  particular 
function;  they  must  possess  a  sense  of  ownership  and  pride  in  the  entire  IBIS  process. 
These  important  elements  of  the  human  interactions  with  the  overall  IBIS  process  can 
only  be  attained  through  continual  training. 

The  training  must  make  a  concerted  effort  to  dispel  any  element  of  fear  in  the  work 
place.  The  human  portion  of  the  process  must  desire  to  work  well,  not  fear  to  work  poorly. 
Towards  the  end  of  dispelling  fear,  the  humans  involved  in  the  system  must  understand 
the  SQC  plan  and  its  purpose.  The  intention  of  the  SQC  plan  is  not  to  identify  poor 
inspectors  and  fire  them.  Its  intention  is  to  help  the  inspectors  perform  their  duties  by 
identifying  problems  with  the  IBIS  process.  Once  identified  these  problems  will  be  fixed. 

The  training  must  also  instill  an  atmosphere  that  elicits  continual  improvements. 
All  people  involved  with  the  system  must  be  an  integral  part  of  the  improvement  planning 
and  implementation.  If  the  human  element  of  the  process  possesses  a  sense  of  pride  for 
the  system,  they  will  seek  out  and  find  ways  to  improve  the  overall  process.  Once  any 
improvements  are  proven  feasible  they  should  be  implemented. 
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B.8  Cost  and  Benefits  Analysis 

Now  that  the  SQC  plan  has  been  detailed  the  question  that  must  be  addressed  is, 
why  do  SQC  at  all?  If  SA-ALC  invests  time  and  resources  for  an  IBIS  SQC  plan,  what 
benefits  can  they  expect?  The  following  cost  and  benefits  analysis  addresses  this  issue. 

B.8.1  SQC  Costs.  The  costs  associated  with  the  SQC  plan  manifest  them  self  as 
both  quantifiable  and  unquantifiable  costs.  Operational  and  capital  costs  represent  the 
quantifiable  costs,  while  human  resistance  to  change  represents  the  unquantifiable  costs. 

In  order  to  establish  an  annual  cost  associated  with  the  SQC  plan  the  operational 
and  capital  costs  must  be  presented.  Personnel  costs  represent  the  primary  operational 
costs  for  the  SQC  plan.  A  single  SQC  inspector  will  meet  the  inspection  requirements 
of  the  IBIS  SQC  plan.  This  is  justified  because  266  blades  require  SQC  inspection  (see 
sectionB.6).  A  blade  should  take,  on  average,  5  minutes  to  inspect  and  measure  cracks. 
The  SQC  inspector  therefore,  can  complete  266  blades  in  22.2  hours.  This  represents  55.4% 
of  an  40  hour  work  week.  There  is  also  a  need  to  perform  FP  processing  and  data  recording, 
and  these  tasks  would  complete  the  SQC  inspectors  work  week.  The  costs  for  the  SQC 
inspector  are  $48.50  per  hour,  and  this  translates  to  $97,000  per  year.  The  SQC  plan  wiU 
also  require  an  NDI  engineer  to  oversee  the  SQC  plan.  The  SQC  plan  calls  for  one-half  an 
engineer  person-year.  The  cost  of  this  NDI  engineer  for  the  SQC  plan  is  $60,000  per  year. 
Therefore,  the  annual  operation  costs  for  the  SQC  plan  is  $157,000. 

The  capital  costs  associated  with  the  SQC  plan  emerge  in  the  purchase  of  a  fluorescent 
microscope  at  $30,000,  a  blade  counter  at  $6,000,  and  SQC  software  at  $2,000.  This  is  a 
single  investment  of  $38,000.  Establishing  an  effective  uniform  annual  cost  (EUAC)  using 
a  10  year  life  cycle  and  a  10  percent  interest  rate  generates  an  EUAC  of  $6,184.  The  final 
effective  uniform  annual  cost  is  $157,000  -b  $6, 184  =  $163, 184  per  year. 

The  unquantifiable  costs  associated  with  the  SQC  plan  emerge  as  human  resistance 
to  change.  The  IBIS  management  and  engineers  must  first  believe  in  the  advantages  of 
an  SQC  plan.  They  must  then  affect  change  in  the  IBIS  process  that  will  allow  random 
sampling,  calibration  experiments,  and  designed  experiments.  This  change  must  occur  in 
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a  manner  such  that  all  personnel  involved  with  the  IBIS  will  understand  the  purpose  of 
the  SQC  plan,  support  it,  and  work  towards  its  success. 


B.8.2  SQC  Benefits.  The  implementation  of  an  SQC  plan  for  the  IBIS  process 
offers  both  quantitative  and  qualitative  benefits.  The  quantitative  benefits  are  those  ad¬ 
vantages  that  can  be  measured  as  a  result  of  the  IBIS  SQC  plan.  These  advantages  are 
measured  by  the  reduction  of  both  the  number  of  good  blades  classified  as  bad  and  the 
number  of  bad  blades  classified  as  good.  The  qualitative  benefits  emerge  from  the  SQC 
plan  as  effective  implementation  of  long  term  improvements  to  the  system,  assuring  the 
IBIS  customers  of  a  quality  product,  and  increased  knowledge  of  the  IBIS  and  its  overall 
process. 


B.8.2. 1  Quantitative  Benefits.  In  order  to  measure  the  quantifiable  benefits 
of  the  IBIS  SQC  plan,  a  cost  must  be  associated  with  a  good  blade  classified  as  bad  and  a 
bad  blade  classified  as  good  (see  appendix  D).  The  cost  of  jet  engine  blades  range  from  $3 
to  $300  and  each  good  blade  rejected  by  the  system  requires  the  purchase  of  a  new  blade. 
The  cost  of  classifying  a  cracked  blade  as  good  is  related  to  the  cost  of  an  unscheduled 
maintenance  event.  In  the  case  that  an  engine  has  a  blade  with  a  crack  of  critical  size,  an 
unscheduled  maintenance  event  will  occur.  Each  unscheduled  maintenance  event  costs  the 
Air  Force  $58,500  (see  appendix  D). 

The  next  question  to  address  emerges  as,  what  is  the  worst  czise  scenario  for  the 
SQC  benefits  analysis?  This  worst  case  occurs  if  the  IBIS  does  not  and  can  not  operate 
in  an  unacceptable  manner.  That  is  the  IBIS  process  will  always  meet  its  performance 
requirements.  Under  this  scenario  the  SQC  plan  will  save  the  Air  Force  nothing.  It 
will  actually  cost  the  Air  Force  $163,000  per  year.  If  the  IBIS  managers  and  engineers 
believe  this  scenario  to  be  true,  and  they  have  evidence  to  support  this  belief,  then  the 
implementation  of  an  SQC  plan  for  the  IBIS  is  not  advisable. 

Information  is  not  available  pertaining  to  the  quality  of  the  current  IBIS  process, 
therefore  an  assessment  as  to  how  often,  the  process  fails  to  meet  its  performance  require¬ 
ments  becomes  infeasible.  These  requirements  are  a  90/95  POD  for  a  .070  inch  crack  for 
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blades  that  are  inspected  solely  by  humans,  and  a  90/50  POD  for  a  .045  inch  crack  for 
blades  that  are  inspected  using  a  automated  or  semi-automated  inspection  system  (45). 
If  the  SQC  plan  were  to  prevent  the  IBIS  process  from  operating  below  these  stipulated 
POD  values,  SA-ALC  would  place  fewer  cracked  blades  in  jet  engines. 

In  order  to  demonstrate  an  advantage  of  implementing  an  SQC  plan  for  the  IBIS  the 
break  even  point  for  annual  cost  to  annual  cost  avoidance  is  analyzed.  The  break  even 
point  is  the  point  were  the  annual  cost  of  the  SQC  plan  is  less  than  the  cost  avoided  by 
not  misclassifying  blades  as  a  result  of  SQC.  The  annual  cost  of  the  SQC  plan  is  $163,000. 
The  Air  Force  will  save  $58,000  for  each  unscheduled  maintenance  event  that  is  avoided 
as  a  result  of  the  SQC  plan.  The  SQC  plan  will  avoid  unscheduled  maintenance  events 
by  identifying  when  the  system  performance  degrades  below  acceptable  levels,  which  will 
subsequently  reduce  the  number  of  bad  blades  placed  in  service  during  this  time  period. 
Once  the  SQC  plan  identifies  a  system  degradation  the  NDI  engineers  can  find  the  problem 
and  fix  it.  Therefore,  the  IBIS  process  will  misclassify  fewer  cracked  blades  becaues  it 
operates  for  less  time  in  a  degraded  mode.  If  the  SQC  plan  avoids  three  unscheduled 
maintenance  events,  it  will  offer  a  return  on  the  investment  of  $163,000.  This  implies  the 
SQC  plan  must  prevent  as  few  as  three  critically  cracked  blades  from  being  placed  back 
into  service  in  order  to  offer  a  return  on  its  investment. 

The  SQC  plan  will  save  the  Air  Force  approximately  $100  for  every  good  blade  that 
does  not  get  rejected  by  the  IBIS  due  to  unacceptable  IBIS  performance.  Therefore,  if 
the  SQC  plan  allows  the  IBIS  to  correctly  classify  1,630  good  blades  that  otherwise  would 
not  have  been  correctly  classified,  the  SQC  plan  will  have  broken  even.  Of  course,  any 
appropriate  breakdown  of  correctly  classified  blades,  good  or  bad,  due  to  the  SQC  plan 
will  insure  that  the  breakeven  point  will  be  achieved. 

A  second  analysis  of  the  quantifiable  benefits  of  the  SQC  plan  illustrates  the  cost 
avoidance  capabilities  of  the  SQC  plan  by  defining  two  feasible  scenarios  and  presenting 
the  resultant  cost  savings  to  the  Air  Force.  This  analysis  examines  only  the  cost  impli¬ 
cations  of  unscheduled  maintenance  events.  The  first  scenario  is  a  minor  degradation  in 
performance.  A  minor  performance  degradation  is  defined  as  a,l%  increase  in  the  system’s 
missed  detection  rate.  The  second  scenario  is  a  major  degradation  in  performance.  A  ma- 
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jor  performance  degradation  is  defined  as  a  5%  increase  in  the  system’s  missed  detection 
rate. 

As  stated  previously,  the  SQC  plan  will  offer  a  cost  savings  by  identifying  unaccept¬ 
able  IBIS  performance  and  initiating  corrections  to  any  related  problems.  In  the  case  of  the 
minor  performance  degradation,  two  or  three  random  samples  will  be  required  to  identify 
such  a  problem.  Since  the  random  samples  occur  twice  a  week,  the  system  with  SQC  could 
perform  in  this  degraded  mode  for  up  to  1.5  weeks.  However,  without  the  SQC  plan  the 
problem  could  remain  undetected  for  an  indeterminate  period  of  time.  If  the  assumption  is 
made  that  the  problem  will  be  solved  by  a  periodic  maintenance  event  or  a  happenstance 
discovery  of  the  degraded  performance  within  a  month,  a  three  week  differential  between 
the  system  with  and  without  SQC  is  under  examination.  The  scenario  establishes  three 
fewer  weeks  of  minor  performance  degradation  under  the  system  with  an  SQC  plan. 

The  cost  avoidance  calculations  utilize  the  cost  of  incorrectly  classifying  a  blade 
model  presented  in  appendix  D.  The  probability  of  identifying  a  crack  (PIC)  values,  which 
are  equivalent  to  one  minus  the  missed  detection  rate  values,  are  varied  from  .9  -  .995  for 
the  undegraded  system.  The  model  is  run  for  these  values  for  a  3  week  period.  The  PIC 
values  were  then  degraded  by  subtracting  .01,  and  the  model  was  run  a  second  time.  The 
related  misclassification  costs  were  subtracted  and  the  raw  cost  avoidance  amount  of  the 
SQC  plan  for  this  scenario  resulted.  The  annual  cost  of  the  SQC  plan,  $163,000  was  then 
subtracted  from  the  raw  cost  avoidance  amount  and  this  value  represents  the  true  cost 
avoidance  of  the  SQC  plan.  Figure  B.4  plots  these  values. 

This  graph  shows  the  cost  avoidance  for  the  different  PIC  values.  As  the  PIC  values 
increase  (ie  the  missed  detection  rates  decrease)  the  cost  avoidance  of  the  SQC  plan  in¬ 
creases.  This  characteristic  shows  that  the  SQC  plan  will  reduce  the  number  of  bad  blades 
accepted  by  the  system,  and  this  reduction  will  have  a  direct  impact  on  the  number  of 
unscheduled  maintenance  events. 

The  minor  performance  degration  curve  crosses  the  0  dollar  cost  avoidance  line,  at  a 
PIC  value  of  .941.  For  the  systems  with  PIC  values  less  than  .941,  the  SQC  plan  appears 
to  cost  more  than  it  is  worth  if  a  single  minor  performance  degradation  occurs  annually. 
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This  is  indeed  the  case.  However,  such  a  system  stiii  calls  for  the  SQC  plan  because  there 
exists  a  large  potential  improvement  in  the  system,  and  the  SQC  plan  would  help  identify 
improvements.  After  all,  The  SQC  plan  still  prevents  the  same  number  of  bad  blades  during 
the  three  week  period  for  the  system  with  low  PIC  values  as  the  system  with  high  PIC 
values.  The  difference  in  cost  avoidance  lies  in  the  number  of  assembled  jet  engines  being 
free  of  cracked  blades.  Since  the  system  with  lower  PIC  values  has  a  relatively  large  missed 
detection  rate  many  engines  are  being  reassembled  with  cracked  blades.  The  improvement 
the  SQC  plan  makes  in  terms  of  identifying  the  degraded  performance  does  not  prevent  the 
unscheduled  maintenance  events  that  would  normally  occur  under  acceptable  performance. 

The  second  curve  on  figure  B.4  plots  the  cost  avoidance  for  a  major  degradation  in 
performance.  The  derivation  is  the  same  as  for  the  minor  performance  degradation  except 
the  major  performance  degradation  is  a  5%  increase  in  missed  detection  rate,  and  the 
amount  of  time  the  system  performs  in  this  degraded  mode  is  shorter.  The  SQC  plan 
would  identify  such  a  large  change  in  performance  in  a  single  random  sample,  and  the 
system  without  the  SQC  plan  should  identify  that  a  problem  exists  in  a  2  week  period. 
This  leaves  a  time  differential  between  the  systems  with  and  without  the  SQC  plan  as 
1.5  weeks.  The  curve  shows  the  similar  results  as  the  minor  performance  degradation 
case.  Drawing  values  from  figure  B.4  for  the  fully  automated  inspection  system  with  SQC, 
yields  a  $1.2  M  cost  avoidance  for  the  minor  performance  degradation  and  a  $1.6  M  cost 
avoidance  for  the  major  performance  degradation. 

B.8.2.2  Qualitative  Benefits.  The  quantitative  SQC  benefits  analysis  pre¬ 
sented  above  does  not  definitively  illustrate  that  SQC  will  save  the  Air  Force  money  in 
terms  of  fewer  misclassified  jet  engine  blades.  However,  the  analysis  demonstrates  what 
must  occur  in  order  for  the  SQC  plan  to  be  justified  in  terms  of  cost.  The  SQC  plan  must 
cause  a  reduction  in  misclassified  blades.  The  analysis  of  quantifiable  benefits  centers  on 
the  identification  and  correction  of  immediate  problems  with  the  IBIS  that  result  in  de¬ 
graded  performance.  The  SQC  plan  can  also  cause  a  reduction  in  misclassified  blades  by 
initiating  systematic  improvements  to  the  IBIS  that  would  decrease  the  missed  detection 
and  false  call  rates.  The  SQC  plan  also  provides  information  to  the  engine  design  organi- 
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zations  that  would  reduce  the  number  of  cracks  in  jet  engine  blades.  These  improvements 
lead  to  a  presentation  of  qualitative  benefits  of  an  SQC  plan. 

The  qualitative  benefits  that  result  from  the  implementation  of  an  SQC  plan  are  the 
effective  implementation  of  systematic  improvements,  an  increase  in  knowledge  pertaining 
to  the  IBIS  process  and  its  product,  and  the  ability  to  assure  the  customer  of  a  quality 
product.  Any  systematic  improvements  to  the  IBIS  will  either  reduce  the  cost  of  operating 
the  system  or  increase  the  inspection  capability.  Also,  increasing  the  amount  of  knowledge 
about  the  system  offers  an  inherent  improvement  provided  that  it  is  used  appropriately. 
In  the  case  of  the  IBIS,  the  knowledge  must  be  used  first  to  close  the  loop  on  the  system 
and  control  the  process  so  it  will  always  meet  its  performance  specifications.  Second,  the 
knowledge  must  be  used  to  effect  systematic  improvements  to  the  IBIS.  Third,  the  IBIS 
must  provide  any  pertinent  information  to  those  responsible  for  engine  design  in  order  to 
address  the  problem  of  cracked  blades  at  the  source.  Finally,  the  information  gained  by 
the  SQC  plan  will  serve  to  assure  quality  to  customers  of  the  IBIS  process. 

B.  9  Conclusion 

The  IBIS  represents  a  production  system  that  accepts  used  jet  engine  blades,  pro¬ 
cesses  them  using  fluorescent  penetrant  inspection  techniques,  and  outputs  a  product  to 
the  operational  commands.  This  product  is  jet  engine  blades  free  of  cracks.  This  pro¬ 
duction  concept  must  remain  in  the  forefront  of  management’s  vision.  The  system  has 
a  customer,  and  the  customer  has  a  right  to  receive  quality  parts.  They  also  have  the 
obligation  to  expect  accountability  for  part  quality. 

The  IBIS  basically  operates  as  an  open  loop  process,  and  this  characteristic  represents 
a  significant  flaw  in  the  IBIS  process.  Blades  enter  the  system.  The  system  processes  and 
inspects  the  blades.  Blades  leave  the  system  as  acceptable  or  flawed.  At  some  point  in 
the  past  the  system  proved  its  functionality  and  now  blind  trust  is  placed  in  the  system’s 
proper  performance.  This  uncontrolled  operation  is  unacceptable  and  an  SQC  plan  must 
be  implemented  to  close  the  loop  on  the  IBIS  process.  Such  a  closed  loop  system  will 
identify  faults  when  they  occur  and  point  towards  improvements  to  the  overall  process. 
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The  primary  end  of  any  statisticaJ  analysis  must  be  a  sound  process  that  produces 
quality  parts.  This  is  an  important  concept  that  must  be  understood  and  conscientiously 
implemented  in  an  SQC  plan.  Note,  the  primary  end  of  the  statistical  analysis  is  not 
the  verification  of  part  quali  y.  An  SQC  plan  can  not  test  quality  into  the  outgoing 
parts.  However,  the  results  of  an  SQC  plan  can  illuminate  deficiencies  and  lead  to  system 
improvements.  The  data  must  be  analyzed,  engineering  judgements  must  be  applied,  and 
then  both  the  statistical  tools  and  the  experience  of  the  engineers  will  identify  system 
failures  and  implement  corrective  actions.  This  will  result  in  improved  quality. 

The  SQC  plan  will  close  the  loop  on  the  IBIS  by  using  information  from  the  output  of 
the  process  to  impact  the  system  process  itself.  The  SQC  plan  seeks  to  identify  failures  and 
problems  in  the  system  so  the  NDI  engineer  can  implement  corrections.  The  SQC  plan  also 
must  aid  the  NDI  engineers  in  the  identification  of  fundamental  improvements  to  the  overall 
IBIS  process.  When  a  system  demonstrates  that  it  meets  standards,  readily  identifies  and 
fixes  problems  in  the  process,  and  seeks  out  and  finds  process  improvements,  it  represents 
a  quality  process.  Through  the  implementation  of  a  comprehensive  and  coherent  SQC  plan 
the  IBIS  will  eventually  represent  the  model  jet  engine  blade  inspection  system  for  the  Air 
Force. 


Appendix  C.  Camera  Experiments 


C.  1  Objective 

The  objective  of  the  experiments  with  a  charge  coupled  device  (CCD)  camera  is 
twofold:  (l)  to  test  the  hypothesis  that  a  CCD  area  scan  camera  can  be  used  as  the 
primary  sensor  for  image  collection  and  (2)  to  look  for  features  of  a  blade  crack  that  are 
not  currently  being  used  for  blade  classification.  The  experiments  were  performed  as  a 
proof  of  concept  and  give  limited  results  due  to  the  small  sample  size  used  and  the  fact 
that  the  cracks  were  all  very  similar. 

C.2  Introduction 

One  of  the  problems  with  the  FPIM  in  its  current  configuration  is  the  need  for  a 
complex  scan  plan  due  to  the  sensor  requirements.  The  array  nature  of  an  area  scan  CCD 
camera  has  the  potential  to  image  a  complete  side  of  a  blade  in  one  exposure  eliminating 
the  need  for  complex  scan  plans.  CCD  cameras  also  have  more  stable  outputs  than  the 
photo-multiplier  tube  (PMT)  currently  being  used  when  temperature  or  supply  power 
fluctuates.  The  camera  used  for  these  experiments  also  has  more  greyscale  resolution  than 
the  current  FPIM  sensorwhich  may  provide  more  information  about  indications  to  more 
accurately  classify  blades. 

C.3  Procedure  Overview 

One  driving  force  behind  this  experiment  is  to  test  the  hypothesis  that  the  fluorescent 
penetrant  (FP)  wicks  out  of  a  crack  differently  than  it  wicks  out  of  non-crack  indications. 
To  support  the  idea  that  the  FP  does  wick  out  of  cracks  in  a  manner  that  can  be  exploited, 
3  images  were  taken  of  5,  stage  9  compressor  blades  furnished  by  SA-ALC.  After  the  blades 
were  prepared  with  water  soluble  FP,  three  images  were  taken  of  each  blade  using  a  liquid 
cooled  CCD  camera.  The  first  image  of  each  blade  was  taken  before  the  developer  was 
applied  to  the  blade  and  the  second  image  was  taken  one  minute  after  the  developer 
was  applied.  The  third  image  was  taken  after  a  variable  dwell  time.  The  images  were 
subtracted  from  each  other  so  the  only  indications  left  in  the  final  image  were  the  result 
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of  the  penetrant  wicking  out  between  the  first  image  and  the  second  image.  The  two 
subtraction  images  were:  (1)  an  image  at  the  longest  dwell  time  minus  the  image  at  one 
minute  of  dwell  time  and  (2)  an  image  at  the  longest  dwell  time  minus  the  image  before 
the  developer  was  applied.  The  other  driving  force  behind  these  experiments  is  to  support 
the  use  of  a  CCD  camera  as  the  primary  sensor  and  exploit  any  inherent  advantages  it 
may  have  over  the  current  sensor.  Onlj'  stage  9  compressor  blades  with  similar  cracks  were 
examined  so  the  results  must  be  further  verified  by  a  much  broader  set  of  blades  before 
any  definitive  conclusions  can  be  reached. 

C.4  Test  Setup 

To  test  the  hypothesis  that  the  FP  wicks  out  of  a  crack  differently  than  it  wicks 
out  of  non-crack  indications,  the  blades  were  prepared  using  Turco  Products,  fluro-chek 
wp  180  LS,  type  I/A/4  fluorescent  penetrant  and  American  Gas  and  Chemical  spray-on 
developer  P/N  AG-DH.  The  penetrant  was  liberally  applied  to  the  blade  using  a  cotton 
swab.  After  application,  penetrant  was  allowed  to  seep  into  discontinuities  for  at  least  40 
minutes.  Following  this  time,  surface  penetrant  was  removed  using  a  pressurized  stream 
of  tap  water.  The  blades  were  then  dried  in  front  of  a  lOOOW  electric  space  heater  for  30 
minutes.  No  developer  was  applied  until  later  into  the  test.  A  batch  of  five  blades  was 
prepared  and  the  blades  not  used  immediately  were  removed  from  the  heater  and  kept  at 
room  temperature  until  needed. 

All  tests  were  conducted  in  a  dimly  lit  room  to  reduce  the  effects  of  ambient  light. 
The  equipment  used  is  listed  in  table  C.l  and  equipment  layout  is  shown  in  figure  C.l. 
The  CCD  camera  used  was  liquid  cooled  to  -45®C  and  implemented  a  576  x  384  array  of 
23/im  X  2Zpm  pixels  with  a  sensitivity  of  at  least  10~®  lux.  For  the  stage  nine  compressor 
blades,  the  lens  was  focused  to  image  an  area  of  2"  X  1.33"  at  a  distance  of  14"  resulting 
in  each  pixel  representing  about  0.0035"  x  0.0035"  on  the  blade  surface.  The  camera  unit 
was  interfaced  to  a  personal  computer  via  a  GPIB  interface  bus  for  data  storage. 


Nomenclature 

Manufacturer 

Model 

CCD  Camera 

Photometries 

Star  I 

100  mm  telephoto  lens 

Sigma 

XQ 

Visible  absorbing  filter 

Pyrex 

7-54 

Sharp  cut  red  filter 

Pyrex 

3-70  filter 

62  mm  Deep  yellow  filter 

Vivitar 

15G 

386  Personal  computer 

Zenith 

ZBO-3303-GQ 

Portable  black  light 

Magnaflux 

ZB-26 

Table  C.l. 

Equipment  List 

Figure  C.l.  Equipment  Setup 
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C.5  Test  Procedure 


Images  were  taken  of  9  stage  9  compressor  blades  before  and  after  developer  was 
applied.  Four  of  the  images  were  not  usable  for  image  subtraction  because  the  images  were 
initially  stored  in  a  file  format  that  corrupted  the  data.  The  camera  exposure  time  for  each 
blade  remained  fixed  between  images.  The  images  after  the  developer  was  applied  were 
taken  at  several  different  dwell  times  for  each  blade.  The  blade  images  were  stored  on  the 
PC  in  a  FITS  format  using  the  software  supplied  by  Photometries.  The  files  were  then 
transferred  to  a  SUN  Sparcstation  2  for  processing  using  the  image  processing  program 
Khoros.  The  images  of  a  single  blade  taken  at  different  times  were  subtracted  from  each 
other  to  identify  significant  changes  in  the  indication.  A  threshold  was  applied  to  the 
subtracted  images  because  of  the  variations  between  the  two  images  that  could  not  be 
controlled  such  as  vibrations  or  light  changes.  The  images  were  then  normalized  to  255 
gray  scales  so  they  could  be  displayed.  The  resulting  images  were  then  examined  in  detail 
for  any  significant  features  of  cracks  that  could  possibly  be  exploited  for  blade  classification. 

C.6  Results 

C.6.1  Gradient  Measurements.  The  data  from  the  CCD  contains  all  of  the  infor¬ 
mation  that  is  provided  in  the  FPIM  data  (pixel  intensity  with  the  location  of  the  pixel)  so 
it  is  reasonable  to  believe  all  of  the  features  to  test  the  implementation  of  a  neural  network 
can  be  extracted  from  the  CCD  data  in  much  the  same  manner  (see  appendix  G). 

The  CCD  has  better  space  and  intensity  resolution  than  the  FPIM  sensor  due  to 
the  characteristics  of  the  CCD  camera  and  the  setup  used  during  the  experiments.  The 
CCD  camera  has  16  bits  of  intensity  resolution  as  compared  to  10  bits  of  resolution  for  the 
FPIM  sensor  which  results  in  65,536  possible  distinct  intensity  levels  for  the  CCD  camera 
compared  to  1024  possible  levels  for  the  FPIM  sensor.  The  experiment  setup  also  resulted 
in  a  space  resolution  of  3.5  mils  in  the  x  and  y  directions  for  the  CCD  camera  compared 
to  8  mils  in  the  x  and  y  direction  for  the  FPIM  sensor.  The  better  resolution  allows  for 
less  quantization  error  when  extracting  features  to  be  used  for  classifying  an  indication. 
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One  feature  that  the  resolution  impacts  is  the  gradient  across  an  indication.  The 
gradient  of  intensity  across  a  crack  indication  seems  to  have  a  very  distinct  pattern  of 
two  peaks  with  a  valley  between  them  when  plotted  across  the  width  of  the  indication 
(see  figure  C.2).  Each  peak  is  on  either  side  of  the  crack  with  the  valley  centered  on  the 
crack  itself.  This  is  partially  due  to  the  penetrant  centered  on  the  crack  fluorescing  at  the 
maximum  possible  intensity.  The  amount  of  penetrant  tapers  as  you  move  away  from  the 
crack  and  the  intensity  of  the  indication  decreases.  The  area  of  maximum  intensity  occurs 
over  an  area  of  several  pixels  so  the  gradient  between  adjacent  pixel  is  near  zero. 


Gradient 

Intensity 


Gradient  profile  from  FPIM  data 


Figure  C.2.  Plot  of  Intensity  Gradient  Across  a  Crack  Indication 


The  plot  of  gradient  using  the  FPIM  data  is  much  less  distinctive  than  the  plot  using 
the  CCD  camera  data.  The  indications  used  for  the  CCD  data  is  not  the  same  indication 
as  the  one  used  to  acquire  the  FPIM  data  due  to  the  unavailability  of  the  blade,  but 
the  indications  were  chosen  because  of  their  similar  appearance.  The  two  indications  are 
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about  the  same  width  and  the  plots  are  aligned  such  that  the  origin  of  each  plot  is  at  one 
edge  of  the  crack.  The  CCD  plot  has  higher  peaks  and  sharper  changes  because  of  its 
better  resolution.  This  more  accurate  information  allows  the  detection  of  smaller  changes 
in  intensity  and  more  samples  across  an  indication.  If  the  valley  between  the  peaks  is 
less  than  two  pixel  widths  across  then  it  could  be  missed  altogether.  By  having  smaller 
pixels  this  feature  could  be  more  reliably  extracted  from  smaller  cracks.  Using  this  more 
detailed  information  to  exploit  the  change  in  the  gradient  across  a  crack  could  lead  to 
better  features  and  better  classification  performance. 

C.6.2  Image  Subtraction.  Five  blades  were  used  in  the  final  analysis  with  the  wait¬ 
ing  times  after  drying  and  longest  dwell  times  shown  in  table  C.2.  Before  the  developer 


Blade 

Number 

Waiting 
Time  (minutes) 

Longest  Dwell 
Time  (minutes) 

5 

3 

10 

6 

88 

20 

7 

158 

14 

8 

20 

19 

smear 

24 

8 

Table  C.2.  Blade  Processing  Times 

was  applied  to  each  blade,  the  part  of  the  blade  with  the  highest  intensity  indications  were 
those  caused  by  glare  from  the  black  light  refelcting  off  of  the  blade  being  received  by 
the  camera.  Attempts  to  eliminate  the  glare  were  made  but  were  unsuccessful,  so  only 
a  portion  of  each  was  imaged  to  eliminate  the  glare  from  the  edges  of  the  blade.  After 
the  developer  was  applied,  the  crack  indication  was  very  bright  and  was  easily  separated 
from  the  glare  which  was  usually  much  less  intense  than  the  indication.  The  developer  also 
caused  the  background  intensity  of  the  blade  to  increase  across  the  entire  blade  which  is 
not  readily  seen  in  the  figures. 

Blade  5  was  the  first  blade  whose  images  were  analyzed  with  the  subtraction  tech¬ 
nique  described  in  section  C.5.  The  original  images  for  blade  5  are  shown  in  figure  C.3. 
After  finding  the  correct  threshold  to  apply  to  the  subtracted  images  (through  trial  and 
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Figure  C.3.  Blade  5:  Images  in  order  are;  no  developer,  1  minute  after  developer,  and  10 
minutes  after  developer. 

error),  the  wicking  of  the  penetrant  can  be  easily  seen  as  a  horseshoe  shaped  indication 
around  the  crack  (see  figure  C.4).  The  dark  center  of  the  horseshoe  shape  is  how  much 
the  penetrant  wicked  out  in  1  minute.  The  image  before  the  developer  was  applied  shows 
a  crack-like  indication  near  the  center  of  the  blade.  That  indication  did  not  remain  after 
the  image  subtraction  and  thresholding  since  the  amount  of  penetrant  that  wicked  out  of 
the  indication  was  insignificant  compared  to  the  amount  of  penetrant  that  wicked  out  of 
the  crack.  This  result  is  reason  for  both  hope  and  concern:  hope  because  the  non-crack 
indications  were  removed  and  concern  becav''  the  possibility  exists  that  a  crack  could  be 
under  that  indication  and  get  subtracted  away  resulting  in  a  missed  crack.  The  glare  also 
seen  in  the  image  taken  before  the  developer  was  applied  is  of  interest  for  the  same  reasons 
as  stated  for  the  non-crack  indication  that  it  may  hide  a  crack.  The  next  steps  taken  were 
to  image  a  blade  with  a  larger  non-crack  indication  to  see  if  it  would  also  subtract  out  and 
to  get  data  on  another  crack  to  support  the  observations  just  made. 

Blade  6  could  not  be  imaged  entirely  without  excessive  glare  so  it  was  put  at  the  edge 
of  the  imaging  area  to  block  the  glare.  Blade  6  also  had  a  non-crack  indication  near  the 
center  of  the  blade.  The  non-crack  indication  on  blade  6  was  brighter  than  the  non-crack 
indication  on  blade  5  (see  figure  C.5).  The  same  image  processing  procedure  was  followed 
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Figure  C.4.  Blade  5:  Images  in  order  are;  blade  after  10  minute  dwell  minus  blade  after  1 
minute  dwell,  and  blade  after  10  minute  dwell  minus  blade  before  developer. 


for  blade  6  as  described  in  section  C.5  and  the  results  were  very  similar  to  the  results  for 
blade  5.  The  crack  indication  grew  over  time  and  was  easily  seen  in  the  subtracted  images 
and  the  non-crack  indications  did  not  grow  and  were  not  seen  in  the  subtracted  images 
(see  figure  C.6). 

Blade  7  was  the  third  blade  used  for  analysis  to  investigate  the  trend  seen  in  blades 
5  and  6.  Blade  7  was  also  imaged  at  the  edge  of  the  look  area  of  the  camera  to  eliminate 
the  glare.  The  same  image  processing  was  performed  as  before  except  the  longest  dwell 
time  used  was  14  minutes  (see  figure  C.7).  The  threshold  used  for  blade  7  was  the  same  as 
used  for  blade  6  since  they  were  imaged  under  similar  conditions.  The  crack  in  blade  7  was 
much  more  pronounced  before  the  developer  was  applied  than  the  previous  cracks  because 
it- was  imaged  3  hours  after  drying  which  allowed  more  penetrant  to  wick  out  of  the  crack. 
The  previous  blades  were  imaged  about  an  hour  and  a  half  after  drying.  Because  of  the 
delay  before  imaging,  the  results  from  the  subtraction  of  the  two  images  taken  after  the 
developer  was  applied  did  not  show  the  crack  as  pronounced  as  the  previous  blades  (see 
figure  C.8).  The  more  interesting  result  was  that  when  using  the  images  taken  before  the 
developer  was  applied  and  14  minutes  after  developer  was  applied,  the  crack  became  very 
obvious  in  the  subtracted  image.  This  result  indicates  that  the  waiting  time  before  the 


C-8 


Figure  C.5.  Blade  6:  Images  in  order  aje;  no  developer,  1  minute  after  developer,  and  20 
minutes  after  developer. 


Figure  C.6.  Blade  6:  Images  in  order  are;  blade  after  20  minute  dwell  minus  blade  after  1 
minute  dwell,  and  blade  after  20  minute  dwell  minus  blade  before  developer. 


C-9 


Figure  C.7.  Blade  7:  Images  in  order  are;  no  developer,  1  minute  after  developer,  and  14 
minutes  after  developer. 


Figure  C.8.  Blade  7:  Images  in  order  are;  blade  after  14  minute  dwell  minus  blade  after  1 
minute  dwell,  and  blade  after  1  minute  dwell  minus  blade  before  developer. 
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developer  does  aflfect  the  results  of  the  subtraction.  However,  if  the  images  are  taken  before 
and  after  developer  is  applied  then  the  effect  of  the  dwell  time  is  greatly  reduced. 

Since  all  of  the  results  are  obtained  from  subtracted  images  and  the  glare  and  pen¬ 
etrant  blob  questions  remain,  blade  8  was  imaged  with  a  large  blob  of  penetrant  on  the 
dovetail,  excessive  glare  on  the  edge,  and  reflections  from  the  area  behind  the  blade  (see 
figure  C.9).  The  same  experiments  were  performed  as  before  except  the  long  development 
time  was  19  minutes.  The  image  resulting  from  the  subtracting  the  second  image  from  the 


Figure  C.9.  Blade  8:  Images  in  order  are;  no  developer,  1  minute  after  developer,  and  19 
minutes  after  developer. 

third  image  produced  no  visible  indications.  The  image  resulting  from  the  subtraction  of 
the  before  developer  image  from  the  19  minutes  after  developer  image  showed  much  more 
information  about  the  crack  (see  figure  C.IO).  Results  seem  very  promising  for  an  image 
subtraction  process,  even  with  the  bright  reflections,  the  extreme  glare,  and  penetrant 
blob. 

To  address  the  concern  of  a  blob  of  penetrant  over  a  crack  masking  the  crack,  a 
blade  was  prepared  with  a  smear  of  penetrant  over  the  cracked  indication.  The  blade  was 
prepared  as  before  and  then  a  smear  of  penetrant  was  made  with  a  cotton  swab.  The  blade 
was  then  put  in  front  of  the  heater  for  drying  a  second  time.  The  blade  smear  completely 


C-11 


Figure  C.IO.  Blade  8:  Blade  after  19  minute  dwell  minus  blade  before  developer. 


covered  the  crack  and  obscured  it  before  developer  was  applied  (see  figure  C.ll).  The  image 
after  just  1  minute  dwell  time  shows  the  crack  clearly  with  the  smeared  area  also  much 
brighter.  The  subtracted  image  of  the  eight  minute  dwell  time  image  minus  the  before 
developer  image  shows  other  indications  that  did  not  subtract  away  (see  figure  C.12). 
This  other  indication  is  due  to  the  penetrant  being  ran  over  the  edge  of  the  blade.  This 
indication  does  not  appear  to  be  a  difficult  problem  in  the  classification  of  this  blade.  The 

I 

biggest  concern  is  that  the  subtraction  to  eliminate  the  smear  could  alter  the  features  of 
the  crack  thus  making  it  harder  to  classify.  This  type  of  a  smear  is  an  extreme  case,  since 
most  blades  are  properly  prepared.  The  blade  was  imaged  after  8  minutes  of  development 
time  and  the  before  developer  image  was  subtracted.  The  resulting  image  clearly  shows 
the  crack  indication  with  a  smaller  but  distinct  blob  seen  previously  (see  figure  C.12).  The 
indication  is  so  small  compared  to  a  crack  that  it  would  not  affect  the  classification  of 
this  blade.  The  'blem  of  penetrant  blobs  on  the  blade  also  seems  to  have  a  solution  by 
subtraction  of  two  images  taken  before  and  after  developer  is  applied. 
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Figure  C.ll.  Smeared  Blade:  Images  in  order  are;  no  developer,  1  minute  after  developer, 
and  8  minutes  after  developer. 


Figure  C.12.  Smeared  Blade:  Images  in  order  are;  blade  after  8  minute  dwell  minus 
blade  after  1  minute  dwell,  and  blade  after  8  minute  dwell  minus  blade 
before  developer. 


C.  7  Conclusions 


The  results  of  these  experiments  are  at  best  preliminary  and  only  suggest  possible 
solutions.  The  data  set  was  very  limited  and  should  be  expanded  in  future  tests.  Even 
though  this  limited  set  contained  only  stage  nine  compressor  blades  with  cracks  at  the  tip, 
the  results  show  that  there  are  characteristics  of  cracks  that  may  be  exploited  for  blade 
classification. 

Using  two  images  of  the  same  blade  and  subtracting  and  thresholding  them  does 
hold  promise  in  rejecting  background  due  to  poor  processing  or  small  surface  defects.  The 
data  provided  here  suggests  using  one  image  before  the  developer  is  applied  and  the  other 
after  allowing  the  penetrant  to  wick  out  after  developer  is  applied  holds  the  best  hope  for 
a  solution. 

The  use  of  a  sensor  with  better  resolution  could  lead  to  better  features  to  classify 
blades.  The  gradient  across  a  crack  is  one  such  feature  that  may  be  exploitable  with  more 
accurate  data. 

The  ability  to  use  these  techniques  described  have  the  potential  to  be  used  for  turbine 
blades.  Although  turbine  blades  were  not  evaluated  in  these  experiments,  the  differences 
in  gradient  across  a  crack  and  the  wicking  rates  of  penetrant  out  of  cracks  and  other 
surface  irregularities  should  be  exploitable  for  turbine  blades  in  much  the  same  way  as  for 
compressor  blades. 

The  CCD  camera  was  able  to  image  all  of  the  blades  and  detect  FP  indications.  It 
worked  under  varying  conditions,  providing  reason  to  believe  that  a  CCD  camera  would 
be  sufficient  for  the  primary  sensor  for  all  types  of  compressor  and  possibly  turbine  blades. 


C-14 


Appendix  D.  Cost  of  Incorrectly  Classifying  Blades 


D.  1  Overview 

Not  detecting  a  cracked  blade  during  inspection  may  cause  jet  engine  damage  which 
has  an  associated  cost.  The  objective  of  this  appendix  is  to  demonstrate  how  the  cost  of 
incprrectly  classifying  a  blade  (CICB)  can  be  quantified.  To  do  this,  the  block  diagram 
shown  in  figure  D.l  is  discussed  in  detail.  Specifically,  a  review  of  applicable  engine  docu¬ 
mentation  is  discussed  in  section  D.2.  Following  this  discussion,  specific  inspection  system 
configurations  aJong  with  needed  assumptions  are  addressed  in  section  D.3.  Next,  the  cost 
of  incorrectly  classifying  blades  model  is  developed  in  section  D.4  and  optimization  of  one 
inspection  system  detailed  in  section  D.5.  Since  several  key  assumptions  were  needed,  a 
sensitivity  analysis  is  presented  in  section  D.6.  To  check  the  model,  a  validation  is  per¬ 
formed  in  section  D.7.  Finally,  section  D.8  details  some  suggested  improvements  that  can 
be  made  to  the  model. 


inputs 


Actuarial  Engine  Removal 


Figure  D.l.  Thought  process  diagram 
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D.2  Documentation 


D.2.1  Overview.  In  order  to  develop  the  CICB  model,  several  key  pieces  of  infor¬ 
mation  must  be  discussed.  In  this  section  a  review  of  the  present  engine  documentation 
is  presented  with  the  objective  of  acquiring  inspection  interval  and  flaw  size  information. 
Armed  with  this  data,  a  discussion  of  severity  events  is  presented  followed  by  the  calcu¬ 
lation  of  the  probability  that  a  severity  event  occurs.  It  is  shown  that  the  probability  of 
an  unscheduled  maintenance  event  is  considerably  higher  than  the  probability  of  weapon 
system  loss,  so  quantifying  the  cost  of  an  unscheduled  maintenance  event  is  evaluated 
in-depth. 

D.2. 2  Applicable  Documentation.  Several  Air  Force  and  commercial  aircraft  have 
been  involved  in  incidents  due  to  structural  failures  as  a  result  of  material  imperfections, 
manufacturing  defects,  or  fatigue  induced  cracks  (14).  To  counter  the  threat  these  de¬ 
fects  have  on  United  States  Air  Force  assets,  the  Air  Force  formed  the  Engine  Structural 
Integrity  Program  (ENSIP).  Military  Standard  1783,  which  covers  the  ENSIP,  provides 
guidance  towards  what  tasks  need  to  be  performed  for  engine  systems  delivered  to  the 
USAF.  These  tasks  include  design  information,  design  analysis,  material  characterization 
and  development,  component  and  core  engine  testing,  ground  and  flight  engine  tests,  and 
engine  life  management.  In  the  course  of  completing  these  tasks,  the  contractor  identi¬ 
fies  engine  critical  components  during  failure  mode  effects  criticality  analysis  (FMECA) 
and  system  safety  hazard  analysis.  Critical  components  are  classified  as  either  fracture 
critical  or  durability  critical.  Fracture  critical  parts  (FCP)  are  those  whose  failure  will 
result  in  probable  loss  of  the  aircraft  as  a  result  of  non-containment  or,  for  single  engine 
aircraft,  power  loss  preventing  sustained  flight  either  due  to  direct  part  failure  or  by  caus¬ 
ing  other  progressive  part  failures.  Durability  critical  i)arts  (DCP)  are  those  who.se  failure 
or  deterioration  will  result  in  a  significant  maintenance  burden  but  will  not  impair  flight 
safety.  Table  D.l  lists  the  blade  criticality  for  the  Fl00-P\V-229  engine,  which  is  a  high 
performance  version  of  the  FlOO  engine  (15). 

D.2. 3  Calculation  of  Inspection  Interval  and  Initial  Flaw  Size.  The  rupture  time 
(RT)  is  the  analytically  predicted  time  that  a  crack  recjuires  to  grow  from  an  initial  flaw 
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Table  D.l.  FlOO-PW-229  criticality  for  F-15  and  F-16  aircraft. 


size  to  a  critical  point.  At  the  critical  point,  extremely  rapid  growth  occurs  resulting  in 
component  failure.  Taking  one-half  the  RT  defines  the  inspection  interval,  as  shown  in 
figure  D.2. 


Initial  flaws  are  assumed  to  be  present  due  to  material  irregularities,  manufacturing 
damage,  or  low  cycle  fatigue.  The  initial  flaw  size  typically  represents  the  low  end  of  the 
flaw  inspection  capability. 

ENSIP  documentation  delivered  with  the  engine  provides  inspection  intervals  and 
initial  flaw  sizes.  Unfortunately,  the  ENSIP  documentation  is  misleading  since  it  implies 
that  flaw  growth  rates  are  calculated  for  all  fracture  critical  parts,  which  is  not  the  case. 
Most  of  the  effort  toward  flaw  growth  is  concerned  with  fracture  critical  disks  and  other 
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high  spin  velocity  components.  To  date,  analytical  calculations  of  flaw  growth  rate  in 
compressor  blades  are  not  performed  (56).  Historical  data,  basic  material  properties,  and 
limitations  of  inspection  techniques  dictate  the  determination  of  the  smallest  detectable 
flaw  size  for  compressor  blades.  Since  any  detectable  crack  may  be  critical  in  certain  zones 
of  a  blade,  the  Air  Force  specifies,  in  the  appropriate  engine  Technical  Order  (TO),  what 
inspection  technique  will  be  performed  and  to  what  level.  For  the  most  part,  this  equates 
to  the  smallest  detectable  flaw  demonstrated  for  a  given  inspection  technique  (64). 

D.2.4  Severity  Class  Events.  If  a  flaw  exists  in  a  blade  during  inspection  and  is 
missed  resulting  in  the  blade  being  reinstalled  into  the  engine,  there  is  a  certain  probability 
the  blade  will  cause  damage  before  the  next  inspection.  The  amount  of  expected  damage 
depends  on  the  platform,  the  engine,  the  blade  stage,  and  the  flaw’  missed.  The  FMECA 
for  the  FlOO-PW-229  engine  lists  the  probabilities  of  a  cracked  blade  causing  a  severity 
event.  For  this  engine,  cracked  blades  may  cause  two  types  of  severity  events.  Type  I  and 
Type  IV. 

A  Type  I  event  is  catastrophic  and  may  cause  death  or  weapon  system  loss.  This 
event  is  characterized  by  blade  fracture  which  may  cause  reduction  in  compressor  efficiency 
and  possible  titanium  fire  and  damage  to  downstream  parts.  If  a  significant  portion  of  the 
blade  separates,  there  is  potential  for  uncontained  failure  and  fire. 

A  Type  IV  event  is  negligible  with  respect  to  weapon  system  damage  and  will  not 
result  in  injury,  property  or  system  damage,  but  may  result  in  unscheduled  maintenance 
or  repair  (46).  This  event  is  represented  by  a  minor  reduction  in  compressor  efficiency  and 
if  the  crack  is  large  enough,  fracture  may  result. 

D.2.5  Probability  of  Missing  a  Crack.  The  probability  of  a  cracked  blade  being  re¬ 
installed  into  an  engine,  P{CB},  is  equal  to  the  product  of  the  probability  the  blade  is 
cracked,  P{C},  and  the  probability  the  cracked  was  missed  during  inspection  P{M D). 

P{CB}  =  P{C}  X  P{.MD}  (D.l) 
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From  figure  D.2,  if  a  blade  with  a  crack  between  the  initial  flaw  size  and  critical 
size  is  undetected,  it  is  unlikely  the  crack  will  grow  to  rupture  (fracture)  prior  to  the 
first  scheduled  inspection.  However,  if  a  blade  wdth  a  crack  larger  than  the  critical  size  is 
missed,  the  result  may  be  a  class  I  or  IV  event.  As  mentioned  previously,  the  FMECA  lists 
probabilities  of  cracked  and  fractured  blades  causing  class  I  and  IV  events.  In  older  FIDO 
engines  it  was  observed  that  stage  4  and  stage  9  blades  showed  a  tendency  to  crack  (6.5). 
Since  the  stage  9  blade  probabilities  in  the  FMECA  are  indicative  of  the  other  blades, 
and  the  present  automated  inspection  system  is  capable  of  inspecting  these  stages,  it  is 
worthwhile  to  evaluate  these  blades  to  compare  alternatives.  For  the  FlOO-PW-229  engine 
used  in  F-15  aircraft,  the  probability  of  a  stage  9  compressor  blade  causing  a  Type  I  event, 
given  that  the  blade  is  fractured,  is  .006.  Concentrating  only  on  events  caused  by  blades, 
the  probability  of  a  stage  9  compressor  blade  causing  a  Type  IV  event,  given  that  the  blade 
is  cracked,  is  1  (accurate  to  the  significant  digits  of  the  data  given)  (46).  The  probability  of 
a  class  I  event  P{I}  caused  by  one  fractured  blade  is  equal  to  the  probability  of  a  severity 
class  I  event  given  the  blade  is  fractured  P{/(/’},  multiplied  by  the  probability  that  the 
blade  fractures  given  that  it  is  cracked  P{F\CB},  times  the  probability  of  a  cracked  blade 
in  the  engine  P{CB}  . 


P{I}  =  P{I\F)  X  P{F\CB]  X  P{CB}  (D.2) 

The  probability  of  a  class  IV,  P{IV),  event  is  equal  to  the  product  of  the  probability 
of  a  class  IV  event  given  the  blade  is  cracked,  P{IV\CB}.  and  the  probability  of  a  cracked 
blade  in  the  engine,  P{CB}. 


P{IV}  =  P{IV\CB}  X  PiCB}.  (D.3) 

Since  there  is  more  than  one  stage  9  blade  per  engine,  it  is  necessary  to  calculate 
the  probability  that  any  one  of  the  stage  9  blades  causes  a  particular  type  event.  For  the 
FlOO-PW-229  engine,  there  are  seventy-four  stage  9  blades.  If  any  one  of  these  blades 
cracks  or  fractures,  there  is  a  corresponding  probability  of  a  severity  event.  Since  we 
are  concerned  with  any  one  of  the  blades  causing  failure,  this  can  be  considered  a  series 
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Table  D.2.  FlOO-PW-229  Compressor  stage  reliability 

system.  The  reliability  function  of  a  series  system  of  n  independent  components  is  given 
by  r(p)  =  Pi  X  p2  X  ■  •  •  X  Pm  where  p  is  the  probability  that  the  component  is  functioning. 
In  this  case,  p  =  I  -  P{  component  is  not  functioning  }  (48).  For  the  whole  stage  the 
reliability  is  given  by  equations  D.4  and  D.5  for  class  I  and  IV  events  respectively,  where 
n  is  the  number  of  blades  in  the  given  stage. 


r{/}  =  (l-F{/})" 

r{IV}  =  (l-P{IV}r 


(D.-l) 

(D.5) 


Using  the  historical  value  for  P{CB}  of  .05  and  assuming  P{F\CB}  is  1,  yields  the  results 
found  in  table  D.2. 

The  reliability  of  stage  9  for  type  I  events  is  acceptable  when  compared  to  typical  Air 
Force  system  reliabilities  of  .95  to  .99.  Of  course  it  must  be  remembered  that  this  reliability 
figure  is  only  for  the  stage  9  subsystem  and  not  the  whole  engine.  The  reliability  of  stage  9 
for  type  IV  events  is  considerably  less  than  that  for  tyi)e  I  events.  This  is  acceptable  since 
a  type  IV  event  is  considered  insignificant  by  the  engine  contractor  (46).  Accepting  the 
risk  of  a  type  I  event  based  on  the  above  discussion,  it  seems  prudent  to  investigate  the 
higher  risk  of  a  type  IV  event. 
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D.2.6  Cost  of  an  Unscheduled  Maintenance  Event  (Class  IV).  A  rough  calculation 
for  the  cost  of  a  scheduled  maintenance  event  requiring  the  removal  of  an  engine  core, 
which  is  necessary  to  inspect  the  compressor  blades,  is  the  number  of  hours  to  remove 
the  core  multiplied  by  the  man-hour  rate  (base  level  costs)  plus  the  depot  cost.  From 
conversations  with  FlOO  engine  specialists,  250  hours  are  required  to  remove  an  engine 
core  (39),  the  average  man-hour  rate  is  $  38  per  hour  (10),  and  the  depot  costs  vary  from 
$  49K  for  core  disassembly,  repair  (including  materials),  and  inspection,  to  approximately 
$  lOOK  for  core  disassembly,  repair  (including  materials),  inspection,  and  upgrade  (28). 
(Depot  performs  upgrades  to  modify  engines  from  1800  to  4000  cycles  between  scheduled 
maintenance  events.)  An  approximation  for  the  cost  of  a  scheduled  maintenance  event  is 
therefore  $  58. 5K  without  an  upgrade  and  $  109.5K  with  an  upgrade.  Throughout  the 
remainder  of  this  appendix,  the  cost  of  an  unscheduled  maintenance  event  will  be  assumed 
to  equal  that  of  the  lower  scheduled  maintenance  event  cost  mentioned  above,  specifically, 
$  58.5K. 

D.2.7  Reliability  for  All  Stages.  In  order  to  compare  alternatives,  it  is  necessary 
to  evaluate  the  probability  of  missing  a  crack  for  any  stage.  As  previously  mentioned,  the 
probability  that  a  cracked  compressor  blade  causes  a  class  IV  event  is  the  same  for  all 
stages.  However,  in  the  previous  analysis  it  was  assumed  that  the  reliability  of  stage  9  is 
acceptable  and  class  IV  events  would  be  the  determining  factor.  This  assumption  must 
be  checked  by  reevaluating  the  reliability  of  the  engine  considering  all  compressor  stages. 
Table  D.3  lists  the  reliability  of  each  stage  using  equation  D.4  and  equation  D.2  with  a 
PIC  of  .9. 

Using  the  series  system  equation  to  calculate  the  reliability  of  all  the  compressor 
stages  results  in  a  reliability  of  .9787.  This  number  fails  in  the  range  of  what  we  would 
tolerate  from  a  complete  system,  but  the  compressor  blades  are  not  the  only  components 
of  this  engine  that  contribute  to  the  probability  of  a  class  I  event.  Bearing  in  mind  the 
assumptions  that  led  to  this  calculation,  namely  the  probability  of  rupture  given  a  blade  is 
cracked  equals  1,  a  PIC  of  .9.  and  only  one  inspection  interval  before  rupture,  this  number 


D-7 


Stage 

FiHF} 

Number  of  blades  per  stage 

Reliability 

4 

0.00015 

52 

1 

5 

0.01 

62 

0.9969 

6 

0.0003 

74 

0.9999 

7 

0.01 

80 

0.996 

8 

0.006 

70 

0.9979 

9 

0.006 

74 

0.9978 

10 

0.006 

82 

0.9975 

11 

0.006 

86 

0.9974 

12 

0.006 

78 

0.9977 

13 

0.006 

84 

0.9975 

Table  D.3.  FlOO-PW-229  Compressor  stage  reliability 

is  tolerable  and  the  assumption  that  class  IV  events  may  be  driving  the  cost  for  missed 
blades  is  still  valid. 

D.2.8  Other  Costs.  In  a  typical  inspection  system  there  is  a  trade-off  between 
missed  detections  and  false  calls.  For  the  present  system  and  our  alternatives,  these  terms 
equate  to  the  probability  of  calling  a  bad  blade  good  and  the  probability  of  calling  a  good 
blade  bad.  The  cost  of  calling  a  bad  blade  good  w'as  previously  mentioned  to  range  between 
$  58. 5K  to  $  109. 5K.  The  cost  of  calling  a  good  blade  bad  is  the  replacement  cost  of  that 
blade  assuming  no  scrap  value.  Replacement  costs  for  a  blade  depend  on  the  stage  and 
can  vary  between  $  3  and  $  300  (65).  In  order  to  proceed,  we  will  assume  a  worst  case 
replacement  cost  of  $  300.  Doing  this  seems  reasonable  from  the  standpoint  that  in  order 
to  reclaim  the  blade  there  are  secondary  costs  of  additional  inspection  and  rework. 

D.3  Original  System  Configurations  and  Assumptions 

The  title  of  this  section  alludes  that  one  or  more  system  configurations  were  changed 
during  the  course  of  this  study.  To  remove  the  suspense,  the  original  alternative  2  was 
optimized  as  shown  in  section  D.5.  Throughout  this  section,  the  original  alternative  2  will 
be  referred  to  as  the  old  alternative  2  to  enforce  the  distinction  between  what  is  presented 
here  and  the  optimized  alternative  presented  elsewhere  in  this  document. 
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Since  the  old  alternative  2  is  indicative  of  current  inspection  systems,  it  is  developed 
here  to  emphasize  the  advantages  and  disadvantages  of  its  specific  configuration. 

D.3.1  Notation.  For  each  inspection  system  evaluated,  the  notation  listed  below 
applies. 

Notation; 

1.  g.g  :  good  blades  called  good 

2.  gJb  :  good  blades  called  bad  (false  calls) 

3.  bJ)  :  bad  blades  called  bad  (correct  detections) 

4.  b.g  :  bad  blades  called  good  (missed  detections) 

5.  m  ;  number  of  blades  into  the  system 

6.  n  ;  number  of  blades  sent  to  the  FPIM  or  ABR 

7.  /c  ;  false  call  percentage 

8.  PIC  :  probability  of  identifying  a  cracked  blade 

In  addition,  a  suffix  may  be  used  to  indicate  the  inspection  source.  For  example,  fc  f 
corresponds  to  the  false  call  rate  of  the  FPIM  and  fem  refers  to  the  false  call  rate  of  the 
manual  inspector. 

D.3.2  Block  Diagrams  of  the  Baseline  System  and  .Alternatives.  Figure  D.3  is  a 
block  diagram  of  the  present  inspection  system. 

From  this  figure,  the  blades  leaving  the  system  can  be  calculated  as 

BladesOut  =  (g.gf  +  b.gf  +  g-gni  +  g-bm)  (D.6) 

and 

BadBlades  =  (bJbm  +  g-bin)  (D-7) 

Where: 

b.gf  =  (.05)  X  (n)  X  (1  -  P/C7) 
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Bad  Blades 
Bftd  BlwkaciIkdSM} 
GoodBlMkictBtdBkd 


Blades  Out 

Good  Biftdu  called  Good 
Bad  Bladei  cailtd  Good 


Figure  D.3.  Block  diagram  of  the  baseline  system 


g.bm  =  (.95)  X  {m  -  n)  X  {f  cm)  +  if  cm)  X  (gJbf) 

b.gm  =  (.05)  X  (m  -  n)  X  {I  -  PICm)  +  (b-bf)  X  (I  -  PlCm) 

gjbf  =  (fcf)  X  .95  X  n 
bJ)J  =  (.05)  X  (n)  X  (P/C/) 

Note  that  by  convention,  Blades  Out  includes  all  the  blades  identified  as  good  by  the 
system  and  Bad  Blades  consists  of  all  blades  identified  as  bad.  Each  of  the  alternatives 
are  variations  of  the  baseline  and  the  above  equations  apply  with  minor  variations.  The 
block  diagrams  for  alternatives  1,  old  alternative  2,  and  alternative  3,  along  with  the 
corresponding  adjustment  in  parameters  to  calculate  the  number  of  blades  leaving  the 
system  are  found  in  figures  D.4,  D.5,  and  D.6. 

D.3.3  Assumptions  used  for  Transfer  Function  Calculations.  In  order  to  calculate 
the  number  of  blades  leaving  the  systems,  several  parameters  must  be  known.  Unfortu¬ 
nately  there  is  a  lack  of  good  data  for  false  call  rates  and  PlCs  for  the  baseline  as  well  as 

each  of  the  alternatives  so  assumptions  had  to  be  made.  Table  D.4  shows  the  PIC  and 
false  call  rates  used  in  calculating  the  associated  number  of  blades  leaving  the  system  as 
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Bad  Blades  Blades  Out 
Bid  Blidei  etlkd  Bid  Bl»de»  ctQed  Good 

GoodBl«fc.c.Ik<)B.d  Bid  Bl«fc.cJl.J  Good 


Figure  D.4.  Block  diagram  for  alternative  1.  To  use  baseline  equations,  set  n  equal  to  0. 


Name 

Baseline 

Alt  1 

Alt  2 

Alt  3 

PIC  Human 

.9 

.9 

.95 

■a 

PIC  Machine 

.99 

NA 

.99 

.995 

FaJse-Calls  Human 

.025 

.025 

.025 

NA 

False-Calls  Machine 

.50 

NA 

.25 

.  ■  _ 1 

.025 

Table  D.4.  Assumptions  used  for  the  cost  of  incorrectly  classifying  blade  calculations 


shown  in  equations  D.6  and  D.7.  Sections  D.3.3.1,  D.3.3.2,  D.3.3.3,  and  D.3.3.4  provide 
justification  for  assumptions  used  for  the  baseline,  alternative  1.  old  alternative  2.  and 
alternative  3  respectively. 

D.3.3.1  Baseline  Assumptions.  For  manual  inspection,  the  FlOO-229  engine 
specification  (45)  requires  a  POD  of  90/95  at  .070  inches.  Martin  Marietta  measured  the 
performance  of  the  IBIS  inspectors  under  an  Air  Force  contract  (51).  Unfortunately,  the 
thrust  of  Martin  Marietta's  effort  was  to  determine  the  preferred  blade  preparation  tech¬ 
nique:  hand  line  or  AFPPM.  Because  of  the  type  of  exi)eriinents  performed,  the  manual 
POD  curves  are  confounded  with  the  preparation  process.  If  we  ignored  the  confounding, 
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Bad  Blades  Oui 


Bui  BUdet  ctBed  Bui 
Good  Bladei  c«Bed6<d 


Good  BUJu  called  Good 
Bad  Blades  called  Good 


Figure  D.5.  Block  diagram  for  alternative  2.  To  use  baseline  equations,  set  n  and  ;?) 
equal. 


the  manual  PIC  was  not  able  to  meet  specifications  (51).  Conversations  with  IBIS  person¬ 
nel  (65)  indicate  they  believe  inspectors  have  a  POD  of  .995.  Since  this  number  cannot  be 
verified,  and  is  probably  optimistic,  the  worse  case  PIC  of  .9  was  used  for  the  analysis. 

False  calls,  or  the  number  of  good  blades  called  bad,  are  seldom  measured.  It  is 
IBIS  policy  to  minimize  the  number  of  bad  blades  out  of  the  system,  so  inspectors  are 
encouraged  to  call  a  suspect  blade  bad  rather  than  good.  The  only  data  available  on  false 
calls  comes  from  a  set  of  experiments  done  on  the  .\FPP.V1  line  during  this  system  study. 
The  human  was  fairly  robust  to  the  e.  perimeiital  varianro  induced  in  comparison  to  the 
FPIM.  Assuming  the  inspectors’  false  call  rate  is  not  due  lo  experimental  variations  yields 
a  false  call  rate  of  .025.  This  false  call  rate  was  u.sed  for  all  manual  inspectors  in  the 
evaluation  of  system  alternatives.  For  the  FPIM.  a  PIC  of  .99  and  false  call  rate  of  .5  are 
used.  The  false  call  rate  is  based  on  .Martin  Marietta's  work  (51)  and  the  present  r  PI.M 
operation.  The  FPIM’s  PIC  is  an  interpolation  of  tlie  POD  curve  given  in  (51). 
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Blades  Out 


Figure  D.6.  Block  diagram  for  alternative  3.  To  use  the  baseline  equations,  set  fcm 
and  podm  equal  to  one  and  let  n  equal  in. 

D.3.3.2  Alternative  1.  For  the  first  alternative  the  same  values  used  for  the 
human  inspector  in  the  baseline  were  used  along  with  the  assumption  that  the  statistical 
quality  control  recommendation  in  alternative  one  will  not  immediately  change  the  PIC  or 
false  call  rates. 

D.3.3.3  Old  Alternative  2.  Based  on  human  factors  analysis  of  the  different 
inspection  configurations,  see  appendix  I,  the  manual  inspector’s  PIC  is  expected  to  in¬ 
crease.  The  initial  analysis  was  run  assuming  a  PIC  of  .95  and  a  false  call  rate  of  .025. 
For  the  automatic  blade  reader,  the  same  PIC  as  used  for  the  FPIM  will  be  used.  .99, 
and  a  false  call  rate  of  .25.  These  values  are  based  on  the  results  of  the  neural  network 
evaluation,  discussed  in  appendix  G,  and  anticipated  improvements  in  the  classification 
ability  of  the  software. 

D.3.3.4  Alternative  3.  There  is  minimal  supporting  data  for  the  PIC  and  false 
call  values  used  for  alternative  3.  The  values  used.  IMC  of  .995  and  false  call  rate  .025. 
are  estimates  on  what  may  be  available  using  improved  neural  networks  or  additional 
information  from  the  FP  indications,  see  appendix  C. 
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D.4  Model 

D.4-1  Introduction.  Having  presented  the  transfer  function  for  each  system  along 
with  the  needed  assumptions  and  cost  values,  the  model  for  the  cost  of  incorrectly  classify 
blades  can  now  be  formalized.  Figure  D.7  shows  a  schematic  of  the  CICB  calculation. 


Engine  Documentation 


Depot  Cost 
Base  Cost 


Probability  blade  is  bad 

Number  of  blades 

SYSTEM  (i) 
PIC 
FC 

Blade  Replacement  Cost 


Figure  D.7.  Block  diagram  for  CICB  calculations  Model 


D.4-2  Discussion.  From  figure  D.7,  several  previously  mentioned  points  are  em¬ 
phasized.  For  example,  from  the  engine  documentation,  a  type  IV  event  (unscheduled 
maintenance)  is  considerable  more  probable  than  a  type  1  event.  This  information  feeds 
the  cost  block  along  with  the  depot  and  base  costs.  The  output  of  this  block  is  fed  into  a 
multiplication  junction  where  the  number  of  bad  units  assembled  is  transformed  to  a  cost. 
The  number  of  bad  units  assembled  is  derived  from  the  number  of  bad  blades  called  good 
output  from  the  transfer  function  block.  This  number  in-turn  is  determined  by  the  specific 
system  configuration  and  the  other  inputs  shown  in  the  figure.  Bad  units  assembled  is 
typically  less  than  the  number  of  bad  blades  called  good  because  of  the  probability  that 
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more  than  one  bad  blade  may  be  re-assembled  into  any  one  engine.  Also  output  from  the 
transfer  function  is  the  number  of  good  blades  called  bad.  This  number  is  multiplied  by 
the  cost  to  replace  a  blade  and  summed  to  the  cost  of  assembled  bad  units  resulting  in  the 
final  cost  of  incorrectly  classifying  a  blade  value. 

D.4-3  Initial  Results.  Using  the  transfer  functions  discussed  in  section  D.3  and  the 
assumptions  shown  in  tables  D.6  and  D.7,  the  results  shown  in  table  D.5  were  obtained. 


Name 

9-9 

b-9 

9-b 

Cost 

Baseline 

1.112  M 

6027 

1480 

27790 

$94.9  M 

1.112  M 

6000 

1478 

28500 

$95.03  M 

Old  Alt  2 

1.133  M 

3570 

1383 

7125 

$83.06  M 

Alt  3 

1.112  M 

300 

295 

28500 

$24.46  M 

Table  D.5.  Initial  calculations  for  the  cost  of  incorrectly  classifying  blades  calculations 


Assumption 

Value 

Number  of  Blades 

1.2  M 

Probability  that  a  blade  is  bad 

.05 

Cost  of  an  unscheduled  maintenance  event 

$58.5K 

Blade  replacement  cost 

$300 

Table  D.6.  System  assumptions  for  the  CICB  calculations 

These  results  show  that  alternative  3  is  approximately  a  factor  of  four  better  than 
the  others.  This  is  expected  when  considering  the  high  PIC  and  low  false  call  rate  used  for 
this  alternative.  Since  alternative  1  and  3  differ  only  in  the  type  of  inspection,  man  versus 


Name 

Baseline 

Alt  1 

Alt  2 

Alt  3 

PIC  Human 

.9 

.9 

.95 

NA 

PIC  Machine 

.99 

NA 

.99 

.995 

False-Calls  Human 

.025 

.025 

.025 

NA 

False-Calls  Machine 

.50 

NA 

.25 

.025 

Table  D.7.  False  call  and  PIC  assumptions  for  the  CICB  calculations 
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machine,  alternative  1  would  yield  the  same  results  if  its  PIC  and  false  call  rate  were  the 
same  as  those  used  for  alternative  3. 

The  baseline  and  alternative  1  yield  approximately  the  same  results.  Again,  this  is 
expected  because  of  the  low  number  of  blades  actually  processed  by  the  FPIM  in  com¬ 
parison  to  the  total  number  of  blades  inspected.  With  a  low  number  of  blades  processed 
by  the  FPIM,  the  baseline  and  alternative  one  become  functionally  the  same  because  the 
human  performs  the  majority  of  the  inspection  task. 

D.  5  Optimization 

Looking  at  the  total  number  of  bad  blades  called  good  in  table  D.5,  it  is  seen  that 
the  baseline  does  a  worse  job  than  just  an  inspector  alone,  even  when  the  FPIM’s  PIC  is 
assumed  to  be  .99.  This  is  a  result  of  feeding  all  the  bad  blades  identified  by  the  FPIM 
to  the  human  for  re-inspection.  An  inspection  system  configured  similar  to  the  baseline 
system  actually  results  in  lower  total  PIC  and  false  call  rates  than  those  occurring  in  a 
system  similar  to  the  one  described  for  alternative  1.  The  baseline  system  provides  two 
opportunities  to  incorrectly  classify  bad  blades  as  good  during  the  inspection  process.  In 
fact,  any  system  with  multiple  inspection  stations  where  all  the  bad  items  identified  by 
the  first  inspection  station  are  passed  to  the  second  station  will  have  total  PIC  and  false 
call  rates  which  are  the  product  of  the  two  stations’  PIC’s  and  false  call  rates.  Configuring 
an  inspection  system  in  such  a  manner  is  advantageous  if  the  objective  is  to  minimize  the 
false  call  rate.  However,  since  the  cost  of  calling  a  bad  blade  good  is  significantly  more 
than  the  cost  of  calling  a  good  blade  bad,  the  logical  conclusion  calls  for  the  minimization 
of  the  number  of  bad  blades  called  good.  One  way  to  perform  this  minimization  is  to  have 
the  ABR  classify  blades  into  three  categories,  as  shown  in  figure  D.8. 

Such  a  system  could  be  developed  by  using  a  neural  network  which  classifies  blades 
as  good,  bad,  or  suspect.  If  it  is  possible  to  classify  most  of  the  bad  blades  as  bad,  then  the 
number  of  bad  blades  sent  ot  the  human  inspector  is  considerably  smaller  than  in  the  old 
alternative  two  and  a  decrease  in  bad  units  assembled  should  be  realized.  .\lso,  with  the 
addition  of  a  third  category,  blades  called  suspect,  the  system  should  have  less  bad  blades 
called  good  by  the  FPIM.  Again,  this  should  decrease  the  number  of  bad  units  assembled. 
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Blades  called  had 


Blades  Called  Good 


Bad  Blades  Blades  Out 


Figure  D.8.  Block  diagram  for  alternative  2a. 


D.5.1  Alternative  2  Transfer  Function.  Using  similar  notation  to  that  used  in  sec¬ 
tion  D.3.2,  the  transfer  function  of  the  three  category  inspection  system  can  be  defined. 
In  figure  D.8,  the  blades  leaving  the  system  can  be  calculated  as 

BladesOut  =  (g.gf  +  b-gfAg-gm  +  g.bm)  (D.8) 


and 

BadBlades  =  (bJbm  +  g.bm  +  b-bf  +  g.bf)  (D.9) 

similar  to  the  previous  transfer  function  calculations.  Because  of  the  addition  of  a  third 
category,  blades  called  suspect,  the  notation  from  section  D.3.2  must  be  modified  slightly. 
The  additional  notation  and  the  assumed  values  for  parameters  of  interest  are  shown  in 
table  D.8. 
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Description 


Variable 


Justification 


PIC  Human 

PICM 

.9 

same  as  baseline 

PIC  Machine 

PICA 

.99 

same  as  baseline 

False-Calls  Human 

FCM 

.025 

same  as  baseline 

False-Calls  Machine 

FCA 

.25 

within  realm  of  neural  network 

Prob  a  blade  is  bad 

PB 

.05 

based  on  historical  data 

Prob  a  blade  is  good 

PG 

.95 

based  on  historical  data 

Prob  good  blade  called  good 

PG-G 

1-FCA 

by  definition 

Prob  good  blade  called  suspect 

PG-S 

(l-PG.G)x.8 

Random  background 

Prob  good  blade  called  bad 

PG-B 

1-PG-G-PG-S 

conservation  of  blade  law 

Prob  bad  blade  called  good 

PB.G 

( 1-PICA  )x.  5 

small  cracks  or  a  lot  of  background 

Prob  bad  blade  called  bad 

PB-B 

PICA 

by  definition 

Prob  bad  blade  called  suspect 

PB-S 

1-PB.B-PG-B 

conservation  of  blade  law 

Table  D.8.  Additional  notation  and  assumed  values  for  alternative  2 

The  justification  column  of  table  D.8  will  now  be  further  explained.  First,  the  PIC 
and  false  call  rate  of  the  human  inspector  along  with  the  PIC  of  the  automatic  blade 
reader  are  the  same  as  used  for  the  baseline  calculations,  therefore  they  are  reasonable 
here  also.  In  addition,  the  false  call  rate  of  the  ABR  of  .25  is  felt  to  be  achievable  with 
current  neural  networks.  Second,  the  percent  of  good  blades  called  suspect  is  based  on  the 
assumption  that  if  the  system  could  not  identify  the  blade  as  good,  it  was  probably  due  to 
background.  Random  background  probably  will  not  appear  as  crack  like  indications,  so  it 
seems  reasonable  that  more  good  blades  would  be  called  suspect  rather  than  bad.  Third, 
the  percent  of  bad  blades  called  suspect  is  based  on  the  assumption  that  if  the  system 
could  not  identify  the  blade  as  bad,  it  might  be  due  to  small  cracks  or  a  lot  of  background. 
Without  further  information,  .5  seems  reasonable.  Finally,  for  all  systems  modeled,  the 
number  of  blades  entering  the  system  must  equal  the  number  of  blades  leaving  the  system, 
since  the  system  can  neither  create  or  destroy  blades. 

D.5.2  Alternative  2,  Resjilts  and  Summary.  Using  MathCAD  and  the  assumptions 
listed  in  table  D.8,  the  cost  of  incorrectly  classifying  blades  was  calculated  for  alternative 
2.  Table  D.9  repeats  the  initial  calculations  from  section  D.4.3  along  with  the  results  for 
alternative  2. 

In  comparison  to  the  results  from  section  D.4.3,  alternative  two  outperforms  every 
other  system  examined  except  alternative  three. 
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Name 

9-9 

b-9 

Total  bad  units 

9-b 

Cost 

Baseline 

1.112  M 

6027 

1480 

27790 

$94.9  .VI 

Alt  1 

1.112  M 

6000 

1478 

28500 

$95.03  M 

Old  Alt  2 

1.133  M 

3570 

1383 

7125 

$83.06  M 

Alt  2 

1.077  M 

330 

295 

62700 

$36.08  .M 

Alt  3 

1.112  M 

300 

295 

28500 

$24.4  M 

Table  D.9.  CICB  results  for  the  baseline  and  alternatives 


D.6  Sensitivity  Analysis 

D.6.1  Introduction.  In  the  previous  sections  several  assumptions  were  made  in  or¬ 
der  to  calculate  the  cost  of  incorrectly  classifying  blades  for  the  baseline  and  alternative 
inspection  systems.  To  understand  the  effects  these  assumptions  have  on  the  resulting  cost 
calculations,  isocost  curves  were  calculated  for  each  alternative  while  varying  the  Proba¬ 
bility  of  Identifying  a  Cracked  blade  (PIC)  and  False  Call  rates  (FC)  for  the  system.  The 
following  sections  discuss  the  three  plots  most  relevant  for  this  study:  FC  versus  PIC,  the 
number  of  good  blades  called  bad  versus  PIC,  and  the  number  of  bad  units  assembled 
versus  PIC. 

D.6. 2  General  Considerations.  As  typical  in  a  systems  engineering  project,  this 
study  was  interested  in  the  comparison  of  alternatives.  A  fundamental  question  to  ask 
when  doing  the  comparison  is  what  should  the  alternatives  be  compared  to?  For  this 
study  the  answer  is  fairly  straightforward  due  to  the  existence  of  the  present  system.  For 
each  of  the  alternatives,  the  cost  of  incorrectly  classifying  blades  was  forced  to  equal  that 
of  the  baseline  system.  To  do  this,  a  PIC  value  in  the  range  from  .9  to  .995  was  selected 
and  then  the  FC  was  calculated  so  the  cost  of  the  alternatives  equaled  the  cost  of  the 
baseline. 

The  baseline  cost  was  calculated  using  a  PIC  for  the  human  inspector  of  .9  and  a 
false  call  rate  of  .025.  If  an  alternative  requires  a  human  inspector,  the  person’s  FC  values 
were  set  equal  to  those  used  for  the  ba.seline. 
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Alternatives  1  and  3  yield  the  same  results  if  evaluated  at  the  same  PIC  and  FC. 
This  result  is  not  surprising  because  each  alternative  only  consists  of  one  inspection  station. 
Since  these  alternatives  yield  the  same  data,  only  one  curve  is  shown  on  the  graphs. 

D.6.3  FC  versus  PIC  Curve.  Figure  D.9  is  a  plot  of  the  false  call  rate  versus  the 
probability  of  identifying  a  crack  for  each  alternative  inspection  system.  These  curves 
demonstrate  the  FC  rate  needed  for  a  given  PIC  so  that  the  specified  alternative  has  the 
same  cost  of  incorrectly  classifying  blades  as  the  baseline. 

These  plots  demonstrate  the  ease  at  which  a  given  alternative  can  be  adjusted  to 
beat  the  baseline.  For  example,  it  is  difficult  to  adjust  the  FC  rate  for  alternative  1  or  3 
to  decrease  their  costs.  This  is  because  these  alternatives’  cost  curves  demand  fairly  low 
values  of  FC  to  maintain  equivalence  with  the  baseline.  On  the  other  hand,  alternative  2 
can  easily  be  improved  to  beat  the  baseline  simply  by  adjusting  alternative  2’s  false  call 
rate  below  those  shown  plotted  on  the  graph. 

It  is  significant  to  note  that  alternative  2  cannot  maintain  the  baseline  cost  at  the 
higher  PICs.  This  is  due  to  the  fact  that  with  these  high  PIC  values,  this  design  would 
need  to  call  more  good  blades  bad  than  there  are  in  the  entire  good  blade  population  to 
maintain  baseline  cost.  This  is  a  significant  result  and  implies  that  if  a  three  category 
inspection  system  is  possible,  using  it  to  screen  out  bad  blades  results  in  a  system  that 
is  guaranteed  to  be  better  than  the  baseline.  .4gain  this  result  is  not  surprising  because 
the  system  effectively  decreases  the  bad  blade  population  sent  to  the  inspector  and  hence 
decreases  the  chance  of  calling  a  bad  blade  good. 

D.6.4  Number  of  Good  Blades  Called  Bad  versus  PIC  Curve.  The  number  of  good 
blades  called  bad  versus  PIC  curve  is  plotted  in  figure  D.IO.  As  before,  the  curves  in  this 
figure  are  isocost  curves  using  the  baseline  as  a  reference.  It  is  interesting  to  note  that 
the  old  alternative  two  maintains  the  smallest  number  of  good  blades  called  bad.  This  is 
consistent  with  the  previous  discussion  of  the  resulting  total  false  call  rate  for  this  system 
but  unfortunalty  is  the  incorrect  attribute  of  the  inspection  systems  to  optimize.  If  cost  is 
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Figure  D.9.  FC  versus  PIC. 


a  driving  factor,  it  seems  more  reasonable  to  minimize  the  number  of  items  that  are  most 
expensive,  the  number  of  bad  units  assembled. 


D.6.5  Number  of  Bad  Units  Assembled  versus  PIC  Curve.  Plots  of  the  number  of 
bad  units  assembled  versus  PIC  are  shown  in  figure  D.  11. The  curves  in  this  figure  are 
opposite  to  the  ones  in  figure  D.IO.  This  makes  sense  considering  the  only  two  items  being 
traded  off  in  the  system  to  maintain  equal  cost  with  the  baseline  are  the  percentage  of 
good  blades  called  bad  (FC)  and  bad  blades  called  good  (1-PIC). 

From  figure  D.ll,  the  old  alternative  2  has  the  highest  number  of  bad  units  assembled 
followed  by  alternative  1  or  .3  and  then  alternative  2. 

As  the  PIC  value  increases  for  an  inspection  system,  the  expected  number  of  bad 
units  assembled  should  decrease.  However,  because  old  alternative  two  sends  all  of  the 
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Figure  D.IO.  Good  blades  called  bad  versus  PIC. 

blades  identified  as  bad  to  the  human  inspector  for  re-inspection,  increasing  the  PIC  of  the 
ABR  •‘'hile  leaving  the  PIC  of  the  human  constant  will  not  make  a  significant  difference 
in  the  total  number  of  bad  engines  assembled.  Ihis  is  a  .->ignificant  result  since  the  current 
inspection  system  has  the  same  inspection  configuration  as  old  alternative  two  and.  based 
on  the  above  analysis,  optimizes  the  inspection  system  incorrectly. 

D.6.6  Summary.  Based  on  the  model  presented  in  section  D.4,  isocost  curves  pro¬ 
vide  insight  into  the  inspection  systems  performance.  Specifically,  in  order  to  outperform 
the  baseline,  the  isocost  curves  provided  insight  into  what  FC  rate  and  PIC  values  are 
needed  to  insure  a  lower  CICB  than  the  baseline.  If  an  inspection  system  performs  in  the 
region  below  its  isocost  curve,  a  savings  over  using  the  baseline  system  will  be  realized. 

Alternatives  1  and  3  were  shown  as  the  .same  curve  demonstrating  that  if  both  systems 
had  the  same  PIC  and  FC  rate  they  would  offer  the  same  advantages  over  the  baseline 
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Figure  D.ll.  Bad  units  versus  PIC. 


system.  A  word  of  caution  is  prudent  at  this  point.  It  is  doubtful  that  the  PIC  of  the 
typical  human  inspector  will  ever  obtain  the  high  PIC  values  show  in  these  curves.  This 
doubt  is  based  on  several  subjective  concerns  including  the  probability  of  a  human  having 
a  bad  day  and  the  boredom  of  the  inspection  task.  These  concerns  are  substantiated  by 
the  fact  that  in  the  past,  the  human  inspectors  POD  curves  typically  demonstrate  less 
than  .9  PIC  at  .070  inches. 

D.7  Validation  of  the  Cost  of  Incorrectly  Classifying  BIndfs 

The  calculations  leading  to  the  Cost  of  Incorrectly  Classifying  Blades  (CICB)  de¬ 
pended  on  several  assumptions  discussed  in  previous  sections.  The  objective  of  these 
calculations  were  to  demonstrate  the  cost  to  the  Air  Force  if  an  inspection  system  with 
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the  specified  PIC  and  FC  rate  were  used.  The  following  sections  describe  an  attempt  to 
validate  these  calculations. 

D.1.1  Overview.  To  determine  the  actual  number  of  unscheduled  maintenance  event  . s 
occurring  during  a  given  year,  engine  removal  documentation  (63)  was  acquired.  From  tliis 
documentation,  unscheduled  maintenance  events  caused  by  malfunction  codes  indicative  of 
cracked  blades  were  identified.  Based  on  the  number  of  blades  inspected,  the  total  number 
of  unscheduled  events  was  scaled  to  reflect  expected  blade  inspection  totals.  This  number 
was  multiplied  by  the  cost  of  an  unscheduled  maintenance  event  yielding  the  total  cost  of 
unscheduled  maintenance  events  due  to  cracked  blades. 

D.7.2  Result.^:.  Table  D.IO  lists  the  total  number  of  unscheduled  and  scheduled 
maintenance  events  for  a  one  year  period  (63)  using  the  assumptions  enumerated  below. 


Cracked  Blades 

Unscheduled 

Scheduled 

62 

379 

150 

Table  D.IO.  In  this  table  cracked  blades  is  the  number  of  unscheduled  maintenance  events 
assumed  to  be  caused  by  cracked  blades. 


1:  All  FlOO  engines  should  be  included  in  the  analysis,  this  includes  engine  models  100/.\. 
100/B/C.  200A.  200C.  220A/C.  220/B.  220/E.  and  220/F. 

2:  How  malfunction  codes  142.  143,  147.  193.  372.  42.").  .'>37.  and  690  are  indicative  of 
cracked  blades. 

3:  The  year  considered,  consisting  of  quarters  91-4.  92-1. 92-2.  and  92-3.  is  representative 
of  a  typical  year. 

To  compare  actuarial  data  calculations  with  the  cost  of  incorrectly  classifying  blades 
(CICB)  results,  both  calculations  mu.st  use  the  same  number  of  blades.  In  the  CICB 
calculations  it  was  assumed  1.2  M  blades  need  to  be  inspected.  From  table  D.IO.  a  total 
of  .529  engines  were  inspected  during  the  given  year,  .\ssuming  742  compressor  blades  per 
engine,  a  total  of  392,518  compressor  blades  needed  to  be  inspected.  Scaling  this  number  up 
to  1.2  M  blades,  and  assuming  a  proportional  inrrea,se  in  unscheduled  maintenance  events 
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because  of  cracked  blades,  yields  190  unscheduled  maintenance  events.  Assuming  llie  cost 
of  an  unscheduled  maintenance  event  is  $  58.5K.  the  AF  could  expect  ot  pay  S  ll.llM  for 
these  events,  plus  an  additional  cost  for  misclassifying  good  blades.  A  comparison  of  the 
expected  cost  for  unscheduled  maintenance  events  due  to  cracked  blades  is  shown  in  table 
D.ll 


Name 

Total  bad  units 

Cost 

Baseline  CICB 

1480 

S86.58  M 

Baseline  Validation 

190 

$11.11  M 

Table  D.ll.  Cost  of  incorrectly  classifying  blades  comparision  of  the  model  and  validation 
data. 


Shown  in  column  two  of  table  D.ll,  the  model  considerably  overestimates  the  ex¬ 
pected  number  of  bad  units  assembled.  In  an  attempt  to  explain  this  overestimation,  a 
reevaluation  of  the  assumptions  in  the  model  is  necessary.  As  means  of  a  review,  the 
assumptions  that  drive  the  calculation  of  the  number  of  bad  units  are  enumerated  below. 


1:  The  probability  of  a  blade  being  bad  is  .05. 

2:  The  probability  of  identifying  a  cracked  blade  for  a  human  inspector  is  .9. 

3:  Cracks  greater  than  .070  inches  will  cause  an  unscheduled  maintenance  event  with  a 
probability  of  one. 

Using  data  acquired  from  SA-ALC,  the  FPIM  and  Iniman  inspectors  actually  found 
.007  of  the  incoming  blades  as  bad  on  average  over  a  four  year  time  period.  If  the  .05 
number  used  in  the  model  is  actually  too  high,  the  model  will  overestimate  the  number  of 
bad  units  assembled. 

The  probability  of  identifying  a  cracked  blade  using  a  human  inspector  is  assumed 
constant  at  .9.  This  assumption  will  also  over  inflate  the  expected  number  of  bad  units 
assembled  because  in  actuality  the  POD  curve  is  a  monotonically  non-decreasing  function, 
with  large  cracks  having  a  larger  probability  of  detection  than  smaller  ones.  If  the  popu¬ 
lation  of  cracks  entering  the  inspection  system  are  actually  larger  than  the  .070  inch  value 
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assumed,  using  a  PIC  value  of  .9  penalizes  the  inspection  system  by  overestimating  the 
number  of  bad  units  assembled. 

Assuming  cracks  greater  than  or  equal  to  .070  inches  cause  an  unscheduled  main¬ 
tenance  event  with  a  probability  of  one  may  over  estimate  the  number  of  bad  engines 
assembled. 

D.7.3  Model  Reevaluation.  Using  the  probability  of  an  incoming  blade  being  bad 
as  .007  and  assuming  the  number  of  blades  to  be  inspected  is  392,518,  to  compare  with 
the  validation  effort,  the  model  estimates  the  number  of  bad  units  assembled  and  their 
corresponding  cost  as  shown  in  table  D.12. 


Name 

Total  bad  units 

Cost  due  to  bad  units 

Baseline  CICB 

213 

$12.46  M 

Alt  1 

213 

$12.46  M 

Alt  2 

15 

$.877  M 

Alt  3 

14 

$.819  M 

Baseline  Validation 

62 

$3.63  M 

Table  D.12.  Cost  of  assembling  bad  units  using  the  probability  of  a  blade  being  bad  as 
.007  and  total  number  of  blades  to  inspect  equal  to  392,518. 

Notice  the  decrease  in  the  ratio  of  the  expected  number  of  bad  units  assembled 
between  the  baseline  and  the  validation  data  in  comparison  to  the  results  listed  in  table 
D.ll. 

D.  8  Recommendations 

There  is  still  a  significant  discrepancy  between  the  model  and  validation  results  as 
shown  in  the  bad  units  assembled  column  of  table  D.12.  To  improve  the  model  further 
would  require  at  least  two  additional  pieces  of  information,  crack  size  distribution  and 
system  POD  data,  that  are  not  presently  available.  With  crack  size  information  and  a 
POD  data,  a  model  that  does  not  over  inflate  the  number  of  bad  units  assembled  could 
be  developed.  As  previously  mentioned,  the  .9  PIC  assumption  is  only  accurate  for  small 
cracks  and  may  severely  penalize  the  system. 
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It  is  very  probable  that  not  all  the  discrepancy  between  the  model  and  validation  is 
due  to  the  model.  The  model  calculates  the  expected  number  of  bad  units  assembled  based 
on  assumed  system  characteristics  and  inputs,  but  the  validation  also  has  some  fundamental 
assumptions.  When  comparing  the  model  and  validation  results  directly  there  is  an  issue 
of  timeliness.  The  model  predicts  the  number  of  bad  engines  assembled  per  year,  but  does 
not  predict  when  an  unscheduled  maintenance  event  will  occur  because  of  the  bad  unit. 
The  validation  data  lists  the  number  of  bad  units  detected  during  a  given  time  period,  in 
this  case  one  year  was  used.  By  comparing  the  data  directly  it  is  assumed  that  all  the 
bad  units  assembled  for  a  given  year  will  manifest  themselves  in  the  summary  of  engine 
removal  data  for  the  same  year.  In  actuality,  there  will  be  a  time  lag  depending  on  engine 
usage  and  the  timeliness  of  reporting  events.  In  addition,  there  is  an  issue  of  the  definition 
of  maintenance.  Depending  on  needed  flight  readiness,  there  is  a  probability  that  not  all 
unscheduled  events  get  reported,  or  in  the  case  of  preparing  for  an  exercise,  an  unusually 
high  number  of  unscheduled  maintenance  events  are  performed.  Because  these  additional 
issues  are  confounded  with  the  validation  data,  care  must  be  exercised  in  using  this  data 
as  truth. 

Finally,  any  model  is  only  as  good  as  the  assumptions  that  are  used.  It  is  obvious 
from  the  differences  in  the  results  shown  in  tables  D.r2  and  D.ll  that  significant  changes 
in  cost  can  be  obtained  by  perturbations  in  the  model  assumptions.  If  a  better  estimate 
of  the  cost  of  incorrectly  classifying  blades  is  desired,  it  is  imperative  that  the  inspection 
system’s  operation  be  quantified. 

D.9  Summary 

For  inspection  systems  of  interest,  the  cost  of  incorrectly  classifying  blades  was  quan¬ 
tified  by  developing  a  model  to  relate  the  expected  number  of  missed  detections  and  false 
call  rates  to  unscheduled  maintenance  events.  Using  actuarial  data,  a  validation  of  the 
model  was  performed.  The  model  over  estimated  the  number  of  bad  units  assembled,  so 
the  driving  assumptions  in  the  model  were  re-exaimed.  Additional  data  was  acquired  that 
indicated  the  assumption  pertaining  to  the  probability  that  a  blade  is  cracked  was  over  in¬ 
flated.  Using  this  new  data,  the  model  and  validation  data  results  are  in  closer  agreement, 
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but  the  model  still  overestimates  the  number  of  bad  units  assembled.  Improvements  to  the 
model  are  suggested  but  require  information  about  the  baseline  inspection  system  that  is 
not  currently  available. 
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Appendix  E.  Design  for  Maintainability 


E.l  Maintainability  Design  Methodology 

E.1.1  Introduction.  A  maintainability  design  methodology  offers  the  system  design 
engineers  a  tool  to  insure  the  availability  of  their  product  will  meet  stipulated  requirements. 
Maintainability  issues  must  be  central  in  the  early  system  definition  phases  of  the  design, 
and  the  importance  placed  on  maintainability  must  continue  through  the  entire  design 
process.  A  system  maintainability  methodology  will  incorporate  maintainability  in  the 
design  process.  It  will  establish  maintainability  goals  and  requirements  for  the  system  in 
accordance  with  the  system’s  performance  requirements.  The  methodology  will  also  define 
a  maintainability  plan  to  be  implemented  once  the  system  becomes  operational.  Before 
developing  the  maintainability  methodology,  a  few  basic  concepts  are  presented. 

Availability  is  a  vital  characteristic  of  any  industrial  machine  or  system.  In  order 
to  insure  efficient  mission  completion,  all  required  systems  must  be  available  on  demand. 
System  availability  encompasses  two  essential  design  issues,  reliability  and  maintainability. 
Availability  is  the  amount  of  time  the  performance  of  the  system  is  acceptable.  There  are 
two  measures  used  to  define  a  systems  availability.  The  system  mean  time  between  failures 
(MTBF)  reflects  the  average  operating  time  between  system  failures,  while  the  system  mean 
time  to  repair  (MTTR)  reflects  the  average  amount  of  time  required  for  repairs  when  the 
system  has  failed.  The  sum  of  MTBF  and  MTTR  represents  the  time  the  system  should 
be  available  for  operation.  Equation  E.l  succinctly  expresses  the  availability  relationship. 


availability  = 


MTBF 

(MTBF  +  MTTR) 


(E.l) 


The  designers  of  any  new  system  must  balance  three  integral  concepts  if  they  expect 
to  create  a  successful  system.  These  three  concepts  are:  1)  system  performance,  which 
is  designing  the  system  to  perform  the  required  task  well,  2)  system  reliability,  which  is 
designing  quality  into  the  system  to  diminish  the  occurrence  of  failures,  and  3)  system 
maintainability,  which  is  designing  the  system  so  that  when  failures  do  occur  they  can  be 
fixed  quickly  and  easily. 
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Fault  diagnosis,  fault  repair,  and  repair  verification  represent  the  three  corrective 
maintenance  actions.  These  maintenance  actions  drive  the  maintainability  measure.  MTTR. 
The  corrective  maintenance  actions,  together  with  the  preventive  mainleuance  re(|uire- 
ments  drives  the  maintenance  burden  of  the  system.  The  quantity  of  maintenance  per¬ 
sonnel  required  and  their  level  of  training  define  the  system's  maintenance  burden.  The 
system  designers  must  accept  the  requirement  to  design  for  a  stipulated  system  availability 
and  to  minimize  the  maintenance  burden, 

E.1.2  Performance  Requirements  and  Maintainability.  As  mentioned  earlier,  the 
system  design  must  balance  system  performance,  system  reliability  and  maintainability. 
Trade-offs  will  become  a  necessary  aspect  of  the  design  procedure.  Therefore,  in  order 
to  establish  a  minimum  acceptable  performance,  a  well  defined  mission  statement  must 
be  adopted  along  with  clear  performance  requirements  for  systems  and  subsystems.  The 
mission  statement  serves  as  an  axiom  for  the  system  design.  If  the  mission  statement  were 
to  be  violated,  the  system  would  cease  to  exist  as  originally  defined  and  the  design  would 
represent  an  entirely  different  system.  Assuming  the  system  complies  with  the  mission 
statement,  if  the  system  or  any  subsystems  fails  to  meet  the  stipulated  performance  level, 
a  system  failure  has  occurred.  Subsequently,  such  a  system  failure  would  require  corrective 
maintenance  action. 

E.1.3  Maintainability  Methodologies.  A  methodology  must  be  established  to  man¬ 
age  the  operational  system  for  both  preventive  and  corrective  maintenance.  The  system 
design  must  incorporate  the  maintainability  methodology  at  all  stages  of  the  design  while 
insuring  the  system  exceeds  the  established  performance  requirements  (12).  At  the  early 
stages  of  the  design,  the  methodology  will  be  broad  and  general;  the  maintenance  re¬ 
quirements  and  metrics  will  be  loose  estimates.  As  the  design  progresses  and  the  system 
design  engineers  iterate  on  the  maintainability  methodology,  requirements  and  metric  de¬ 
tails  will  emerge  and  confidence  will  grow  in  the  feasibility  of  attaining  the  maintenance 
and  availability  goals. 


The  maintainability  methodology  also  establishes  the  maintenanre  plan  during  sys¬ 
tem  design.  This  maintenance  plan  represents  the  operational  procedures  that  strive  to 
control  system  failures  and  easily  correct  them  once  they  occur. 

E. 1.3.1  Reliability  Centered  Maintenance  and  Corrective  Mainienanc(  Anal¬ 
ysis.  Moss  presents  a  methodology  for  preventive  and  corrective  maintenance  (41).  Relia¬ 
bility  Centered  Maintenance  (RCM)  and  Corrective  Maintenance  Analysis  (CMA)  together 
establish  a  comprehensive  maintenance  design  procedure.  RCM  is  ‘"the  methodology  to 
implement  the  principle  that  no  preventive  maintenance  task  will  be  performed  unless  it 
can  be  justified.  The  criteria  for  Justification  are  safety,  availability,  and  cost  effectiveness 
in  deferring  or  preventing  a  specific  failure  mode.”(41)  The  RCM  method  has  two  distinct 
parts.  First  the  failure  mode  analysis  establishes  the  implications  of  all  the  foreseeable  fail¬ 
ure  modes.  Second,  the  preventive  maintenance  analysis  identifies  the  costs  and  benefits 
of  preventive  maintenance.  CMA  is  the  methodology  to  “identify  the  specific  corrective 
maintenance  tasks  to  be  performed  for  each  failure  mode.  CMA  must  also  determine  the 
disposition  to  be  made  of  faulty  items  replaced  in  the  course  of  correcting  the  failure  - 
are  they  to  be  repaired  or  discarded .”(41)  These  two  schemes  are  based  on  a  question  and 
answer  approach  to  all  failure  modes  and  failure  sources.  The  RCM  and  CMA  logical  flows 
are  presented  in  figures  E.l  and  E.2  (41). 

An  assembly  hierarchy  must  first  be  defined  for  the  system.  This  hierarchy  will 
graphically  breakdown  the  system  to  subsystems,  subassemblies,  and  eventually  down  to 
piece-parts  which  are  components  that  will  not  be  repaired  on  site.  The  design  engineers 
must  establish  the  system  and  subsystem  performance  requirements.  Then  they  must 
identify  failure  modes,  failure  sources  and  failure  rates  for  the  system  components.  RCM 
and  CMA  must  illustrate  the  impact  of  failures  as  well  as  the  cost  of  preventive  and 
corrective  maintenance.  As  the  design  progresses,  the  maintainability  methodology  will 
solidify  as  an  integral  part  of  the  system. 

E.l. 3. 2  Procedure  4-  A  maintainability  procedure  that  can  serve  as  an  alter¬ 
native  to,  or  in  conjunction  with,  the  RCM/CMA  methodology  is  presented  as  Procedure  4 
of  the  MIL-HDBK  472  Maintainability  Prediction.  “This  procedure  is  based  on  the  use  of 
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Figure  E.l.  RCM  logical  flow 
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operattonal  function 

(end  item  dievibution  and  l^OT^ 

Figure  E.3.  Procedure  4  flow  block  diagram 

historical  experience,  subjective  evaluation,  expert  judgement,  and  selective  measurement 
for  predicting  the  down  time  of  a  system/equipment.” ( 12)  An  availability  time  prediction 
as  well  as  preventive  and  corrective  task  time  predictions  will  result  from  Procedure  4  upon 
the  completion  of  the  system  design.  The  initiation  of  the  maintenance  time  prediction 
requires  the  following  information.  1)  system  block  diagram,  2)  functional  flow  diagram, 
3)  subsystem  block  diagrams,  4)  subsystem  flow  diagrams,  5)  end  item  lists,  6)  end  item 
failure  rates,  7)  a  maintenance  concept,  8)  maintainability  goals,  9)  operational  resources 
(facilities,  personnel,  support  equipment,  etc),  10)  a  detailed  description  of  the  task  be¬ 
ing  performed,  11)  location  at  which  the  task  is  being  performed,  and  12)  environmental 
constraints.  See  figure  E.3  (12)  for  the  procedural  flow  block  diagram. 

E.1.4  Discussion.  The  RCM/CMA  methodology  and  Procedure  4  represent  two 
programs  that  can  be  implemented  in  the  system  design  to  insure  maintainability  will  be 
an  integral  part  of  the  final  system.  They  are  not  the  only  methods  capable  of  achieving 
this  design  goal.  RCM/CMA,  Procedure  4,  and  any  other  acceptable  inEiintainability 
design  plan  that  the  design  engineers  have  familiarity  with  should  receive  consideration.  It 
is  imperative  that  a  maintainability  procedure  be  implemented  at  the  early  stages  of  the 
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system  design.  This  plan  must  balance  system  performance  and  system  reliability  with 
system  maintainability.  A  deliberate  design  will  insure  compliance  with  both  performance 
and  availability  requirements. 

E.2  IBIS  Maintainability  Design 

E.2.1  Introduction.  The  IBIS  Upgrade  Study  represents  the  preliminary  design 
stages  of  the  second  generation  automated  blade  reader  (ABR).  A  sound  maintainability 
methodology  must  be  introduced  at  this  early  stage  of  the  design.  Subsequent  stages  of 
the  design  must  incorporate  a  consistent  maintainability  methodology  as  a  central  aspect 
of  the  system  design.  The  IBIS  Upgrade  Study  utilizes  the  appropriate  aspects  of  both  the 
RCM/CMA  and  Procedure  4  of  MIL-HDBK  472  to  initiate  a  maintainability  methodology 
for  upgrades  to  the  IBIS. 

E.2. 2  IBIS  Performance.  The  first  step  for  the  maintainability  methodology  calls 
for  a  clear  and  concise  mission  statement  for  the  system  to  be  designed.  The  IBIS  mission 
statement  is:  the  IBIS  shall  inspect  jet  engine  blades  for  cracks  and  shall  identify  only 
those  blades  without  cracks  as  fit  for  continued  service.  The  IBIS  must  inspect  jet  engine 
blades  for  cracks  before  it  attempts  to  take  on  any  other  functions. 

Together  with  the  system  mission  statement,  the  maintainability  methodology  calls 
for  system  level  performance  requirements.  The  IBIS  shall  inspect  1,200,000  turbine  and 
compressor  blades  as  well  as  500,000  other  jet  engine  parts  for  cracks.  The  IBIS  shall 
detect  a  .070  inch  crack  with  at  least  a  90%  probability  of  detection  at  a  95%  confidence 
level  (90/95  POD).  This  POD  function  is  a  detailed  statistical  analysis  tool  designed  for 
nondestructive  inspection  for  crack  defects  (4).  The  system  performance  requirements  es¬ 
tablish  parameters  for  acceptable  performance  of  the  system.  They  may  require  refinement 
and  additional  performance  requirements  may  emerge  as  necessary. 

The  maintainability  methodology  also  calls  for  the  identification  of  subsystem  per¬ 
formance  requirements.  The  AFPPM  shall  apply  the  fluorescent  penetrant  system  in  a 
manner  such  that  .070  inch  cracks  shall  be  detected  with  at  least  a  90/95  POD.  The  sig¬ 
nal  to  noise  ratio  for  blades  processed  by  the  AFPPM  shall  be  greater  than  a  stipulated 
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Figure  E.4.  IBIS  assembly  hierarchy 


quantitative  or  qualitative  level.  The  inspection  system  shall  detect  fluorescent  penetrant 
indications  that  represent  .070  inch  cracks  at  the  required  90/95  POD.  The  missed  detec¬ 
tion  rate  describes  the  probability  of  accepting  a  defective  blade  and  shall  be  less  than  10%. 
The  false  call  rate  describes  the  probability  of  rejecting  an  acceptable  blade  and  shall  be 
minimized.  Refinements  to  these  subsystem  performance  requirements  will  emerge.  Also, 
performance  requirements  for  each  component  in  the  assembly  hierarchy  must  be  defined 
as  the  system  design  progresses. 

E.2.3  Assembly  Hierarchy.  The  next  step  in  the  maintainability  methodology  is 
the  formation  of  the  assembly  hierarchy.  The  assembly  hierarchy  is  partially  attainable  at 
this  early  stage  of  the  design.  Figure  E.4  illustrates  the  assembly  hierarchy.  The  IBIS  is 
at  the  highest  level  of  the  assembly  hierarchy  and  represents  the  system  being' designed. 
The  second  level  is  the  subsystem  level.  The  third  and  subsequent  levels  are  the  first  as¬ 
sembly  level,  the  second  assembly  level,  and  third  assembly  level.  The  assembly  hierarchy 
presented  by  Figure  E.4  ends  at  the  third  or  fifth  level,  but  as  the  design  progresses  this 
assembly  hierarchy  must  become  quite  detailed.  Each  branch  of  the  hierarchy  must  end  at 
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what  Moss  defines  as  a  piece-part.  “A  part  (or  piece-part)  is  any  hardware  item  that  can¬ 
not  be  disassembled  into  subord.nate  components  without  disrupting  permanent  physical 
bonds. ”(41)  To  add  to  this  definition,  a  piece-part  can  be  perceived  as  any  hardware  item 
that  cannot  or  will  not  be  disassembled  further.  Any  level  of  the  assembly  hierarchy  can 
contain  piece-parts.  For  example,  looking  at  the  fifth  level  of  Figure  E.4  under  the  ins[)ec- 
tion  system,  ABR,  and  computer,  the  third  assembly  identified  as  a  memory  card  could 
represent  a  piece-part.  The  memory  card  would  be  a  piece-part  if  once  the  maintenatice 
team  identifies  a  failure  in  the  card  the  most  cost  effective  action  dictates  the  return  of  the 
part  to  the  vendor  and  the  replacement  with  a  new  part. 

E.2.4  Failure  Mode  Analysis.  Once  each  piece-part  has  a  corresponding  perfor¬ 
mance  requirement,  the  failure  mode  analysis  can  be  performed.  Tlie  MTBF  for  each 
piece-part  must  be  established  as  well  as  potential  causes  for  the  piece-part's  failure  to 
attain  its  performance  requirement.  In  other  words,  the  expected  life  of  each  piece  part 
must  be  estimated  as  well  as  potential  causes  for  failure.  Failure  modes  for  each  level  of 
the  assembly  hierarchy  can  then  be  compiled  as  f«ulures  of  the  respective  branches  to  the 
next  lower  level.  The  critical  failure  modes  must  be  identified.  These  critical  failures  are 
failures  that  can  cause  a  great  deal  of  physical  damage  to  the  system,  or  they  are  tlx*  fail¬ 
ures  that  are  expected  to  occur  relatively  often.  The  maintenance  plan  must  address  the 
critictil  failures  with  particular  care.  The  attainment  of  the  system  availability  requirement 
is  vital  for  the  success  of  the  system  design.  The  better  the  failure  mode  analysis  the  more 
certain  the  design  team  will  be  in  meeting  the  availability  goal.  The  attainment  of  the 
availability  goal  translates  to  a  sound  system  design. 

E.2.5  Block  Diagrams.  Procedure  4  calls  for  system  and  subsystem  block  diagrams 
and  functional  flow  diagrams.  The  IBIS  Upgrade  Study  represents  the  system  definition 
design  stage  of  the  second  generation  ABR.  At  this  early  stage  of  the  design  the^efinition 
of  initial  system  block  diagram  and  functional  flow  diagrams  is  appropriate.  Figure  E..5 
illustrates  the  system  block  diagram.  Appendix  A  presents  the  functional  flow  diagram. 

The  maintainability  analysis  calls  for  the  continued  breakdown  of  the  subsystems  and 
their  components  until  piece  parts  emerge.  The  engineers  responsible  for  the  development 
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Figure  E.5.  IBIS  system  block  diagram 


of  the  maintenance  plan  must  possess  an  in-depth  understanding  of  the  entire  system 
composition  and  functional  flow.  This  understanding  will  insure  a  complete  failure  mode 
analysis,  and  it  will  serve  as  a  vital  tool  to  develop  a  coherent  fault  diagnosis  plan. 

E.2.6  Maintainability  Goals.  The  maintainability  methodology  calls  for  the  estab¬ 
lishment  of  maintainability  goals  early  in  the  system  design.  The  following  four  main¬ 
tainability  goals  have  been  identified:  minimize  the  system  maintenance  burden,  modular 
design,  self  diagnostic  capability,  and  standardized,  off  the  shelf  components. 

The  IBIS  Upgrade  Study  calls  for  the  minimization  of  the  system  maintenance  bur¬ 
den.  The  system  maintenance  burden  is  the  number  of  people  required  to  maintain  the 
system  and  their  level  of  training.  The  immediate  end  of  this  goal  is  to  reduce  i)ersonnel 
costs.  More  importantly,  this  goal  will  force  designers  to  define  a  readily  maintainable 
system.  If  the  maintenance  plan  calls  for  preventive  and  corrective  maintenance  actions 
that  are  easy  and  quick,  the  maintenance  team  will  not  be  burdened  with  continual  time 
consuming  tasks,  nor  will  they  require  extensive  training  or  background. 

This  goal  also  presses  the  designers  to  attain  the  system  performance  requirement 
with  the  least  comi>lex  design.  A  complex  system  would  require  a  large  maintenance 
burden.  This  is  not  to  say  the  design  must  be  simple,  but  the  designers  must  have  good 
reason  to  introduce  a  complex  subsystem  that  will  require  a  specific  expertise  to  maintain. 


E-9 


The  IBIS  Upgrade  Study  calls  for  a  modular  design.  The  system  hardware  must 
be  composed  of  functionally  compartmentalized  components.  These  components  arc  the 
hardware  modules  and  represent  self  contained  hardware  units  designed  to  perform  a  par¬ 
ticular  function.  If  a  function  fails,  the  suspect  module  will  be  checked  together  with  both 
the  modules  that  provide  its  inputs  and  the  modules  that  use  its  output.  The  mainte¬ 
nance  team  will  make  appropriate  replacements  until  the  system  functions  within  normal 
parameters. 

The  IBIS  Upgrade  Study  calls  for  the  implementation  of  self  diagnostic  capabilities. 
The  system  should  possess  the  ability  to  sense  subsystem  and  component  failures.  This 
capability  becomes  particularly  important  for  the  critical  failure  modes.  Once  the  failure 
is  detected  any  appropriate  action  must  automatically  be  initiated  in  order  to  minimize 
further  damage.  The  ABR  master  computer  should  perform  this  built  in  test  function. 
This  self  diagnostic  capability  should  be  self  contained  whenever  possible,  thus  reducing  the 
amount  of  external  equipment  required  to  maintain  the  system.  Fault  diagnosis  represents 
a  significant  portion  of  the  maintenance  actions,  therefore,  a  system  with  a  high  level  of 
self  diagnostic  capability  will  ease  the  maintenance  burden. 

The  IBIS  Upgrade  Study  calls  for  standardized  components.  This  requirement  will 
directly  impact  the  MTTR  as  well  as  system  operating  costs.  Standardized,  off  the  shelf, 
components  reduce  the  MTTR  measure  because  the  parts  are  readily  available  through 
multiple  vendors.  The  standardized  component  maintainability  goal  will  also  beneficially 
impact  operating  costs.  When  a  particular  vendor's  price  increases  unacceptably,  a  second 
vendor  could  be  sought.  This  competition  will  reduce  the  cost  of  spare  parts.  This  main¬ 
tainability  goal  has  a  third,  intangible  benefit.  Thv  system  survival  will  not  be  directly 
linked  to  a  single  source  for  spare  parts.  If  a  vendor  ceases  the  production  of  a  necessary 
component,  other  vendors  would  address  the  need. 

Through  the  fulfillment  of  the  system  maintainability  goals,  the  design  team  insures 
a  maintainable  system.  A  maintainable  .system  will  significantly  impact  the  systen\  avail¬ 
ability.  A  maintainable  system  will  also  cost  less  to  operate.  One  final,  but  important, 
note  remains.  The  maintainability  goals  must  be  applied  to  the  system  .software  whenever 
applicable.  Software  maintenance  represents  a  growing  problem  in  system  maintenance. 
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E.2.7  Potential  Maintenance  Actions.  The  maintenance  plan  represents  a  portion 
of  the  maintainability  methodology.  This  plan  presents  a  detailed  description  of  the  preven¬ 
tive  maintenance  the  system  will  require  as  well  as  any  foreseeable  corrective  maintenance 
actions.  At  this  early  design  stage  the  maintenance  plan  will  be  limited.  A  basic,  illustra¬ 
tive  listing  of  potential  maintenance  actions  follow.  This  preliminary  presentation  should 
serve  to  initiate  a  comprehensive  maintenance  plan. 

1.  preventive  maintenance  actions: 

(a)  lubrication  of  manipulator  subsystem 

(b)  calibration  testing  of  manipulator  subsystem 

(c)  calibration  testing  of  sensor  subsystem 

(d)  artificial  failures  to  exercise  the  built  in  test  capability 

(e)  routine  inspection  and  replacement  of  degradable  substances 

(f)  visual  inspection  of  critical  components 

2.  corrective  maintenance  actions: 

(a)  established  procedure  for  computer  module  repair 

(b)  sensor  replacement 

(c)  manipulator  component  replacement 

(d)  comprehension  of  self  diagnostic  functions 

(e)  ability  to  diagnose  all  potential  failures  not  monitored  by  the  built  in  test  ca¬ 
pability 

(f)  standard  verification  procedures  that  insure  successful  repair 

E.2.8  Conclusion.  The  design  of  the  next  generation  ABR  must  incorporate  main¬ 
tainability  goals  and  requirements.  Two  primary  reasons  emerge  that  support  design  for 
maintainability.  First,  a  maintainable  system  costs  less  to  operate,  and  second,  a  main¬ 
tainable  system  has  increased  availability.  The  IBIS  Upgrade  Study  pre.sents  RCM/CMA 
and  Procedure  4  as  two  viable  maintainability  methodologies  that,  if  applied,  will  improve 
the  maintainability  of  any  subsequent  system.  The  study  then  presents  initial  maintain¬ 
ability  issues  in  accordance  with  the  presented  design  methodologies.  The  effort  to  develop 
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the  next  generation  ABR  should  consider  RCM/CMA  and  Procedure  4.  but  any  coi)i])re- 
hensive  maintainability  design  methodology-  would  be  appropriate.  Prior  to  development. 
SA-ALC  must  define  performance,  reliability  and  maintainability  requirements.  If  the  next 
generation  ABR  performs  well,  is  reliable,  and  is  maintainable,  it  will  be  successful. 


Appendix  F.  IGRIP  Robotic  Simulation 


F.  1  Introduction 

Two  robot  simulations  were  developed  to  estimate  system  speed.  One  simulation 
included  a  vertically  articulated  manipulator  with  similar  characteristics  to  the  Motoman 
model  K6SB.  The  other  simulation  included  a  Selective  Compliance  Assembly  Robot  Arm 
(SCARA)  manipulator  with  similar  characteristics  to  the  GMFanuc  model  A510.  The 
simulated  workcells  consisted  of  a  manipulator  with  gripper,  a  CCD  camera,  a  conveyor 
system,  a  basket  of  48  blades,  and  an  empty  basket.  Figure  F.l  shows  the  general  workcell 
layout  used  for  the  simulations.  The  manipulators  were  programmed  to  iteratively  select 
one  blade  from  the  basket  of  blades  and  present  the  blade  to  the  camera.  After  the  camera 
simulated  imaging  one  side  of  the  blade,  the  manipulator  rotated  the  blade  180  degrees, 
then  the  camera  simulated  imaging  the  other  side  of  the  blade.  After  both  sides  were 
imaged,  the  manipulator  either  placed  the  blade  back  in  the  original  bzisket,  simulating 
a  good  blade,  or  placed  the  blade  into  the  originally  empty  basket,  simulating  a  suspect 
blade.  The  suspect  blades  were  dispersed  throughout  the  basket  to  obtain  an  average  travel 
distance.  After  the  simulations  were  run,  the  average  time  to  process  one  blade  through 
the  ABR  was  calculated  for  use  in  later  analysis. 

F.2  Simulation  Variables 

Both  the  simulation  workcell  layouts  are  configured  to  be  compatible  with  the  current 
basket  and  conveyor  design  dimensions.  The  blades  used  in  the  simulation  were  scale 
models  of  the  stage  9  compressor  blade.  The  simulation  variables  for  the  manipulators  are 
listed  in  Table  F.l.  Since  the  majority  of  the  manipulator  motion  involved  relatively  short 
travel  distances,  the  simulation  was  sensitive  to  the  values  that  were  used  for  maximum 
joint  acceleration.  As  the  acceleration  values  increased,  there  was  a  decrease  in  the  time 
required  to  inspect  a  basket  of  blades  (simulation  cycle  time).  Vendor  estimates  of  the  joint 
acceleration  values  were  based  on  the  amount  of  time  required  for  the  joints  to  reach  their 
maximum  velocities  which  were  called  acceleration  constants.  Dividing  the  maximum  joint 
velocity  by  the  acceleration  constant  gives  an  estimate  of  the  joint  acceleration,  assuming 
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Figure  F.l.  Workcell  Layout 


linear  acceleration.  Vendor  estimates  of  the  acceleration  constants  were  not  available  in 
some  cases  and  qualified  guesses  in  other  instances.  The  estimates  ranged  from  1.8  seconds 
to  0.288  seconds.  In  an  effort  to  resolve  the  inconsistencies  and  the  lack  of  information, 
the  simulation  estimated  the  required  joint  accelerations  for  each  simulation  to  produce  an 
average  inspection  time  of  7  seconds  per  blade  with  a  25  percent  false  call  rate. 

For  simulation  simplicity,  the  vertical  prismatic  joint  of  the  A510  was  modeled  as 
joint  1  when  it  is  actually  joint  2  on  the  A510.  Using  joint  1  as  the  prismatic  joint  allowed 
the  use  of  a  generic  SCARA  manipulator  readily  available  in  the  software  library. 

F.3  Results 

The  simulations  were  run  to  determine  the  time  required  to  inspect  a  basket  of  blades 
with  fal&»  call  rates  of  0,  2.5,  5,  10,  15.  20  and  25  percent.  Evaluating  the  cycle  time  for 
different  false  call  rates  was  important  since  the  alternative  designs  using  automated  blade 
readers  used  different  projected  false  call  rates.  Using  the  variables  identified  in  Table  F.l, 
the  following  results  were  obtained  (see  tables  F.2  and  F.3). 

Comparing  the  average  inspection  times  per  blade  for  each  system  does  not  indicate 
a  significant  difference  in  the  speeds  of  the  two  manipulators.  Since  the  A510  has  higher 
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K6SB 

A510 

VERT.  ART. 

SCARA 

Joint  Travel  Limit 

Joint  1 

±170  deg 

±304  mm 

Joint  2 

—220  to  20  deg 

±181  deg 

Joint  3 

—45  to  225  deg 

±181  deg 

Joint  4 

±180  deg 

±181  deg 

Joint  5 

±135  deg 

±181  deg 

Joint  6 

±200  deg 

±181  deg 

Max.  Jt  Velocity 

Joint  1 

110  deg/s 

600  mm/s 

Joint  2 

90  deg/s 

Joint  3 

110  deg/s 

270  deg/s 

Joint  4 

240  deg/s 

540  deg/s 

Joint  5 

240  deg/s 

540  deg/s 

Joint  6 

400  deg/s 

540  deg/s 

Max.  Jt  Acceleration 

Acceleration  Constant 

1.0  sec 

0.72  sec 

Joint  1 

Joint  2 

Joint  3 

Joint  4 

Joint  5 

Joint  6 

Table  F.l.  Summary  of  Simulation  Variables 


joint  velocities  and  a  lower  acceleration  constant,  it  would  seem  that  it  would  be  faster 
than  the  vertically  articulated  design,  but  the  data  indicate  the  opposite.  A  more  detailed 
analysis  of  the  simulation  indicated  that  the  majority  of  the  manipulators’  movements 
were  made  in  the  vertical  direction.  The  manipulators  move  in  the  vertical  direction  at 
least  two  times  for  every  blade  in  the  basket.  The  K6SB  can  use  a  combination  of  two 
joints  to  move  vertically,  while  the  A510  must  rely  solely  on  one  joint.  Consequently,  the 
combined  speeds  of  the  K6SB’s  joints  enables  it  to  move  faster  vertically  than  the  A510, 
making  the  K6SB  faster  on  average  during  the  manipulation  of  the  blades.  Additionally, 
the  acceleration  constants  used  may  not  be  accurate  estimates  of  their  true  values.  If  the 
acceleration  constant  for  the  A510  is  significantly  smaller  than  0.72  seconds,  the  results 


F-3 


A510 

False  Call  Rate  (percent) 

0 

2.5 

5 

20 

2.') 

Number  of  Suspect  Blades 

0 

2 

3 

5 

8 

10 

12 

Total  Cycle  Time  per  Basket  (min) 

5.30 

5.34 

5.36 

5.40 

5.47 

5.51 

Avg  Inspection  Time  per  Blade  (sec) 

6.63 

6.68 

6.70 

6.76 

6.83 

6.88 

6.90 

Table  F.2.  Summary  of  Simulation  Results  for  A510 


K6SB 

False  Call  Rate  (percent) 

0 

2.5 

5 

10 

15 

20 

Number  of  Suspect  Blades 

0 

2 

3 

5 

8 

10 

12 

Total  Cycle  Time  per  Basket  (min) 

5.11 

5.19 

5.20 

5.26 

5.37 

5.44 

5.45 

Avg  Inspection  Time  per  Blade  (sec) 

6.39 

6.49 

6.50 

6.58 

6.72 

6.80 

6.82 

Table  F.3.  Summary  of  Simulation  Results  for  K6SB 


would  change.  Figure  F.2  plots  the  average  time  required  to  inspect  a  blade  against  the 
acceleration  constant  for  a  system  with  a  25  percent  false  call  rate.  This  figure  can  be  used 
to  approximate  the  performance  of  a  manipulation  system  based  on  a  25  percent  false  call 
rate  given  the  systems’  true  acceleration  constant. 

Based  on  the  simulation  results  and  the  increased  flexibility  offered  by  a  vertically 
articulated  design  to  manipulate  blades  longer  than  10  inches,  the  K6SB  was  chosen  for 
use  in  the  models  for  alternatives  2  and  3.  During  the  simulation  the  two  baskets  were 
placed  12  inches  apart.  Increasing  the  separation  distance  to  28  inches  resulted  in  little 
change  to  the  A510’s  average  inspection  time  per  blade,  but  increased  the  K6SB’s  by 
approximately  0.5  seconds.  It  is  expected  that  as  the  lateral  distance  is  increased,  the 
A510’s  ability  to  move  quickly  horizontally  will  have  a  greater  effect  on  the  amount  of  time 
to  inspect  a  blade. 

F.4  Summary 

The  simulation  demonstrated  that  both  types  of  manipulators  are  feasible  designs 
for  manipulating  stage  9  compressor  blades  which  indicates  the  feasibility  of  using  com¬ 
mercially  available  manipulators.  Based  on  the  simulation  results  and  the  added  flexibility 
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Average  blade  inspection  time  versus  acceleration  constant 


0  0.5  1  1.5  2  2.5 

.Acceleration  Constant  (sec) 


Figure  F.2.  Average  inspection  time  per  blade  versus  acceleration  constant  at  a  25  per¬ 
cent  false  call  rate 


offered  by  a  vertically  articulated  design,  the  K6SB  was  chosen  for  alternatives  2  and  3 
data  collection.  Analysis  indicated  that  the  A510  may  be  a  more  suitable  choice  if  the  ac¬ 
celeration  constants  used  during  the  simulation  are  significantly  different  from  the  actual 
values  or  if  lateral  distance  between  the  two  baskets  increases  significantly. 
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Appendix  G.  Image  Processing  and  Classification  Software 


G.  1  Introduction 

The  image  processing  and  classification  software  is  a  critical  component  of  any  auto¬ 
mated  blade  inspection  system.  The  quality  of  the  software  will  determine  the  effectiveness 
of  the  system  in  terms  of  how  well  it  identifies  good  and  defective  blades.  The  overall  ob¬ 
jective  of  this  portion  of  the  design  study  was  to  demonstrate  that,  compared  to  the  present 
system,  an  inspection  system  that  uses  more  advanced  image  processing  and  classification 
techniques  can  offer  improved  performance.  The  software  developed  at  this  stage  supports 
design  alternatives  2  and  3,  which  use  automated  inspection  systems.  The  results  of  this 
investigation  also  provide  a  first  approximation  of  the  new  system’s  performance,  which 
allows  for  a  more  complete  characterization  of  the  alternative  2  design. 

The  image  processing  and  classification  software  successfully  demonstrated  that  im¬ 
provements  could  be  made  to  the  existing  system,  wh’^'h  uses  a  simple  rule-based  decision 
process  to  identify  cracks  in  blades.  The  available  data  set  of  fluorescent  penetrant  (FP) 
indications  was  small  and  potentially  contained  errors  because  of  the  way  it  was  labeled, 
but  it  still  allowed  the  performance  of  the  software  to  be  tested.  A  total  of  11  features 
were  collected  from  each  indication  and  used  to  train  a  multilayer  perceptron  network  to 
classify  each  indication  as  good  or  defective.  The  network  achieved  an  approximate  error 
rate  of  21.5%  for  false  calls,  with  a  corresponding  missed  detection  error  rate  of  .5.5%. 
These  values  represent  a  significant  improvement  over  the  present  system,  which  has  a 
false  call  error  rate  of  50%  or  higher  in  the  current  configuration  and  an  unknown  missed 
detection  rate,  which  was  has  been  estimated  at  1%  (65). 

The  following  discussion  describes  the  process  for  developing  the  image  processing 
and  classification  software.  The  primary  steps  include  collecting  and  labeling  data,  se¬ 
lecting  features  that  characterize  crack-type  indications,  developing  software  to  collect  the 
features,  training  a  neural  network  to  classify  indications,  and  interpreting  the  network 
results  using  Bayesian  decision  analysis.  The  conclusion  offers  a  list  of  recommendations 
on  how  research  should  continue  to  further  improve  the  software. 
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G.2  The  FPIM  Baseline  and  Blade  Data 


The  Fluorescent  Penetrant  Inspection  Module  (FPIM)  serves  as  the  baseline  system 
to  compare  the  results  to,  as  well  as  a  platform  to  collect  blade  data.  In  its  present 
configuration,  the  FPIM  performs  very  poorly  and  has  a  false  call  error  rate  of  at  least  50%. 
based  on  data  from  1991.  This  high  error  rate  requires  that  a  human  inspector  look  at  more 
blades  than  necessary,  because  the  FPIM  cannot  accurately  distinguish  between  a  crack 
indication  and  a  non-crack  indication.  For  this  discussion,  an  indication  is  a  continuous 
region  of  fluorescent  penetrant  (FP)  on  the  blade  surface.  In  the  FPIM  data,  pixels  less 
than  two  units  apart  are  considered  connected  and  indication  regions  containing  many 
pixels  are  grown  accordingly. 

G.2.1  FPIM  Image  Processing  and  Classification.  The  FPIM  software  is  written 
in  Fortran  and  is  highly  machine  dependent  because  of  the  way  data  is  stored  and  ma¬ 
nipulated.  The  image  processing  and  classification  software  receives  raw  pixel  data  from 
the  sensor  and  determines  if  the  indications  on  the  blade  surface  are  significant  enough 
to  reject  the  blade.  The  system  is  very  accurate  when  labeling  blades  that  do  not  have 
defects.  In  other  words,  if  the  FPIM  labeled  the  blade  as  good,  there  is  approximately 
a  99%  probability  that  the  blade  is  actually  good.  If  a  blade  has  a  large  indication  or  many 
small  indications,  the  FPIM  memory  buffers  can  fill  and  the  system  may  not  be  able  to 
make  a  determination  within  the  allowed  15  second  time  window.  Since  the  memory  and 
processing  speed  limitations  prevent  the  system  from  accurately  classifying  the  blade,  it 
labels  the  blade  suspect  by  default,  so  a  human  inspector  will  check  it. 

Even  when  the  pixel  data  is  within  the  memory  limits  of  the  FPIM  processor,  the 
system  still  has  trouble  accurately  identifying  cracks.  The  main  problem  is  that  the  decision 
function  uses  a  rule  based  classification  scheme  that  only  considers  the  number  of  pi.xels 
and  the  maximum  pixel  intensity  in  each  indication.  The  original  designers  selected  these 
two  features  based  on  empirical  data  that  showed  most  cracks  could  be  identified  with 
this  limited  amount  of  information  (51).  Cracks  and  other  types  of  flaw  indications  create 
complex  FP  patterns  on  the  blade  surface.  A  human  inspector  uses  a  variety  of  information 
about  the  crack  indications  to  accurately  identify  them.  Indication  size  and  inteiiaity  are 
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important,  but  the  shape  of  the  indication,  how  linear  the  indication  is,  and  the  way  the 
intensity  of  the  indication  changes  across  the  surface  are  all  important  factors  that  help 
distinguish  cracks  from  non-crack  indications.  A  system  that  extracts  more  information 
from  each  indication  and  then  processes  the  information  efficiently  to  make  a  decision,  will 
likely  be  able  to  more  accurately  distinguish  between  good  and  defective  blades  compared 
to  the  FPIM. 

Since  it  requires  information  about  two  indication  features,  the  FPIM  only  performs 
a  small  amount  of  image  processing.  The  present  image  processing  operations  include 
dynamic  background  removal  and  thresholding  to  remove  noise  and  dim  pixels  from  the 
data  set.  The  dynamic  background  operation  applies  a  31  x  31  Gaussian  convolution 
operator  to  the  data  surrounding  each  pixel.  The  values  in  the  convolution  operator  are 
determined  by  the  blade  type  and  the  pixel  location  on  the  surface  of  the  blade.  The  result 
of  the  convolution  is  subtracted  from  the  center  pixel.  See  the  technical  report  describing 
the  IBIS  for  more  information  about  the  dynamic  background  removal  process  (9). 

In  addition  to  dynamic  background  removal,  the  system  applies  a  threshold  uniformly 
to  all  data.  The  threshold  level  was  determined  empirically  as  the  value  that  gave  the 
system  the  best  performance  in  terms  of  the  false  call  rate.  With  a  fixed  threshold,  the 
software  has  no  way  to  compensate  for  blades  that  may  have  slightly  different  average 
intensity  levels. 

The  system  binds  pixels  together  and  grows  regions  which  represent  separate  indica¬ 
tions.  The  FPIM  also  eliminates  indications  with  fewer  than  four  pixels  from  the  data  set, 
because  they  are  too  small  to  be  significant.  The  baseline  image  processing  routines  must 
be  enhanced  to  collect  additional  features  about  the  indications,  which  can  then  lead  to  a 
more  accurate  classification  function. 

G.2.2  The  FPIM  Data  Set.  An  advanced  automated  blade  reader  may  be  able  to 
distinguish  among  cracks,  scratches,  pits,  dents,  and  background  in  FP  indications.  Since 
the  objective  of  this  analysis  was  to  prove  the  concept  behind  alternative  2,  the  design 
team  chose  to  reduce  the  scope  of  the  problem  to  just  identifying  crack  indications.  The 
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personnel  at  SA-ALC  agreed  that  this  would  be  a  reasonable  first  step  for  a  demonstrai  ion. 
In  addition,  this  limitation  reduced  the  time  required  to  collect  and  label  the  data. 

To  develop  new  image  processing  and  classification  software,  San  Antonio  Air  Lo¬ 
gistics  Center  (SA-ALC)  established  a  data  set  of  blade  indications  that  characterized  a 
subset  of  the  type  of  crack  defects  the  new  system  should  be  able  to  identify.  The  only  data 
available  for  this  project  was  pixel  information  collect  by  the  FPIM.  The  data  described 
each  pixel  by  its  intensity  on  a  scale  of  0  to  1023  and  by  its  position  on  the  blade  surface 
in  terms  of  [a:,  j/]  location.  SA-ALC  provided  data  on  189  stage  nine  compressor  blades. 
The  data  was  delivered  in  two  formats:  paper  printouts  from  the  FPIM,  which  diagramed 
the  position  of  the  indications  on  the  blade  surface,  and  in  an  electronic  format  containing 
the  location  and  intensity  information  for  each  pixel. 

The  data  set  contained  a  number  of  problems  which  may  have  distorted  the  results 
of  the  new  software.  Although  imperfect,  the  data  should  be  sufficient  to  make  some 
reasonable  estimates  about  how  the  software  can  perform.  The  first  problem  with  the  data 
set  was  that  it  was  not  directly  from  the  sensor  and  the  FPIM  had  already  completed  some 
preprocessing.  The  preprocessing  included  data  reduction,  dynamic  background  removal, 
and  thresholding. 

The  electronic  data  were  not  labeled  to  indicate  which  pixels  belonged  to  crack 
indications.  The  data  printouts  showed  the  crack  locations  as  identified  by  inspectors,  but 
SA-ALC  collected  the  printout  data  and  the  electronic  data  at  separate  times,  so  the  two 
sets  did  not  match  exactly.  The  blades  were  actually  reprocessed  through  the  Automated 
Fluorescent  Penetrant  Preprocessing  Module  (AFPPM)  between  collecting  the  hardcopy 
and  electronic  data.  The  AFIT  design  team  used  the  printouts  and  electronic  data  to 
estimate  where  the  crack  indications  were  located.  This  is  a  possible  source  for  error,  since 
trained  inspectors  did  not  label  the  electronic  data  used  to  develop  the  new  software. 

Another  potential  problem  with  the  data  set  was  that  it  was  only  from  one  type  of 
compressor  blade.  In  addition,  every  blade  was  defective,  which  meant  that  it  contained 
at  least  one  crack.  This  type  of  data  set  is  clearly  biased  toward  containing  cracks,  which 
occur  infrequently  during  the  FPIM’s  actual  operation.  The  new  software  processed  indi- 
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cations  individually,  so  it  may  not  matter  that  all  the  blades  are  defective  if  the  non-crack 
indications  on  defective  blades  are  representative  of  non-crack  indications  on  good  blades. 
A  better  data  set  would  contain  several  types  of  blades  as  well  as  some  blades  that  were 
not  defective. 

The  problems  with  the  data  set  may  decrease  the  performance  of  the  new  software, 
but  they  are  not  overwhelming  and  should  not  significantly  impact  the  results.  Since  there 
is  a  clear  understanding  of  the  limitations  of  the  data,  the  problem  areas  can  be  avoided. 

G.3  Image  Processing  and  Feature  Extraction 

The  objective  of  image  processing  software  is  to  extract  features  from  the  FPIM  data 
which  can  be  used  to  train  a  neural  network  to  differentiate  between  good  and  defective 
blades.  The  raw  data  from  SA-ALC  must  first  be  converted  into  a  format  that  is  easy 
to  work  with.  Once  the  data  has  been  prepared,  the  features  can  be  measured  from 
each  indication  to  build  training  and  evaluation  sets  for  the  classification  portion  of  the 
software.  Figure  G.l  shows  the  overall  process  for  image  processing  and  classification. 
Image  processing  is  represented  by  the  first  row  in  the  figure. 

G.3.1  Data  Preparation.  The  initial  preprocessing  involves  separating  the  data  files 
for  each  blade  into  data  files  for  each  swath  of  the  FPIM  sensor.  Blades  typically  require 
four  swaths  by  the  sensor  to  completely  cover  the  surface.  The  swaths  overlap  to  avoid 
misclassifying  indications  that  e.xist  near  the  swath  boundaries.  Each  swath  should  be 
processed  independently  to  identify  defects. 

Each  original  data  file  contained  the  information  for  one  blade  and  the  swath  in¬ 
formation  was  separated  within  the  file.  The  design  team  developed  a  BASIC  program 
to  divide  raw  FPIM  data  into  individual  swath  data  files.  The  BASIC  language  was  the 
easiest  to  use  for  this  task  because  it  involved  reading  and  writing  character  strings  and 
searching  for  key  words  in  the  original  files.  From  the  original  189  files  of  blade  data, 
the  program  generated  716  swath  files  of  data.  Most  blades  had  four  swaths  per  file,  but 
several  had  less,  because  of  the  way  the  FPIM  collected  the  data.  Out  of  the  available 
swath  data,  the  design  team  selected  the  first  140  swaths  to  use  for  feature  extraction.  The 
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Figure  G.l.  Flowchart  for  image  processing  and  classification. 


number  of  swaths  was  limited  because  of  time  constraints.  Tiiese  swath  data  files  serve  as 
the  fundamental  form  of  input  for  the  remainder  of  the  image  processing. 

With  the  swath  data  files  established,  the  next  pre])aration  step  was  to  label  the  crack 
indications  in  each  file.  The  design  team  used  a  simple  MathCAD  plotting  routine  to  read 
the  swath  files  and  view  the  pixel  locations.  The  intensity  information  was  not  used  at  this 
stage.  The  graphical  display  of  the  electronic  data  wa.s  compared  to  the  data  printouts 
for  each  swath  to  locate  crack  indications.  The  printouts  showed  the  relative  location  of 
indications  on  the  blade  surface.  A  trained  SA-ALC  inspector  had  examined  the  printouts 
and  the  actual  blades  and  then  circled  the  indications  caused  by  cracks  in  each  printout. 
For  each  swath,  the  number  of  cracks  and  the  [j:,  y]  position  of  each  crack  in  swath  data 
was  labeled.  The  140  swath  files  contained  a  total  of  18;i  crack  indications.  .4s  mentioned 
above,  the  labeling  process  is  a  potential  source  of  error,  since  trained  inspectors  did  not 
label  the  electronic  data. 
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Only  one  position  that  was  within  the  crack  indication  region  need  to  be  labeled,  since 
the  feature  extraction  software  would  be  able  to  match  the  label  to  the  whole  indication.  In 
other  words,  if  one  pixel  within  an  indication  was  labeled  as  belonging  to  a  crack,  then  the 
software  would  label  everything  in  that  entire  indication  region  as  a  crack.  Every  n'gion 
that  is  not  labeled  as  a  crack  is  considered  a  non-crack  indication.  With  the  swath  files 
separated  and  the  crack  indications  labeled,  the  data  was  now  in  an  appropriate  format  to 
collect  information  about  each  indication. 

G.3.2  Feature  Extraction.  The  softw'are  must  collect  features  that  allow  it  to  dis¬ 
tinguish  between  crack  and  non-crack  indications.  As  a  starting  point,  it  makes  sense 
to  use  the  features  collected  by  the  present  system:  the  number  of  pixels  in  the  indica¬ 
tion  (size)  and  the  maximum  intensity  in  an  indication.  These  two  features  provide  some 
useful  information,  but  the  feature  set  was  expanded  to  include  others  that  may  contain 
more  information  about  cracks.  The  new  features  should  more  fully  capture  the  shape 
and  intensity  characteristics  of  crack  indications.  For  this  project,  three  different  types 
of  additional  features  were  collected  from  each  indication:  simple  shape  measurements. 
Hough  transform  features,  and  gradient  features. 

G.3.2. 1  Filtering  and  region  growing.  Before  collecting  any  features,  the  data 
in  each  swath  file  had  to  be  processed  to  identify  and  label  indication  regions.  For  each 
swath,  the  software  read  in  the  number  of  crack  indications  in  the  swath,  the  [x,  y]  location 
of  each  crack,  and  the  complete  list  of  pixel  location  and  intensity  information.  The 
software  applied  a  median  filter,  which  removed  some  of  the  noise  in  the  data  without 
distorting  the  essential  shapes  of  the  indications.  A  median  filter  examines  each  pi.xel  and 
its  immediate  neighbors  within  some  specified  distance.  For  this  project,  a  distance  of 
one  was  used,  so  each  pixel  was  compared  to  the  eight  surrounding  pixels.  The  filter  then 
replaces  the  intensity  of  the  center  pixel  with  the  median  intensity  of  the  set.  of  pixels 
under  examination.  Median  filtering  is  a  standard  image  processing  technique  described 
in  several  texts  (22,  44). 

The  median  filter  effectively  removes  noise,  but  it  also  alters  the  intensity  of  each 
pixel.  For  this  project,  as  much  of  the  original  intensity  information  as  possible  was 
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maintained  to  characterize  the  crack  indications.  With  this  in  mind,  the  filtered  ima^e 
was  used  to  reduce  the  number  of  pixels  in  the  original  image.  The  new  image  data 
contained  a  pixel  at  a  particular  location  only  if  both  the  original  and  filtered  images 
contained  pixels  in  that  same  location.  The  operation  is  similar  to  a  logical  AND  operator 
applied  to  each  image,  pixel  by  pixel.  The  pixel  intensity  in  the  reduced  data  set  was  set 
equal  to  the  intensity  value  in  the  original  data. 

The  software  then  used  the  reduced  data  set  to  bind  regions  of  pixels  together  into 
indications.  The  binding  operation  searched  through  the  data  array  for  pixels  and  con¬ 
nected  together  those  that  were  within  two  pixels  of  each  other.  The  search  continued 
until  all  pixels  were  labeled.  Once  the  pixel  regions  were  identified,  the  software  used  the 
crack  location  information  to  label  the  indications  that  represented  cracks. 

With  the  indication  regions  identified,  the  software  could  now  collect  features  from 
each  indication.  As  mentioned  above,  the  two  features  collected  by  the  FPIM  were  mea¬ 
sured  first:  the  number  of  pixels  and  the  maximum  pixel  intensity  in  each  indication. 
Indications  that  contained  fewer  than  four  pixels  were  eliminated  from  the  feature  set, 
since  they  were  too  small  to  represent  cracks.  The  software  to  collect  the  new  features  was 
then  developed. 

G.3.2.2  Simple  shape  and  intensity  measurements.  The  first  new  set  of  fea¬ 
tures  involved  simple  measurements  about  the  shape  and  intensity  of  each  indication.  The 
software  determined  the  size  of  the  smallest  rectangle  that  could  be  drawn  around  the 
indication.  For  simplicity,  the  sides  of  the  rectangle  aligned  with  the  axes  of  the  pixel 
array.  The  software  then  measured  the  length,  width,  and  length-to-width  ratio  of  the 
surrounding  rectangle.  In  addition,  the  software  calculated  the  average  intensity  of  j)ixels 
within  each  indication.  These  four  features  were  readily  available  from  the  data  set  with 
minimal  programming  effort. 

G.3.2.3  Hough  transform  features.  The  simple  length  and  width  measure¬ 
ments  do  not  capture  enough  information  about  indications  to  distinguish  cracks  from 
other  types  of  indications.  One  feature  that  a  human  inspector  uses  is  the  linearity  of  the 
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indication,  since  cracks  tend  to  be  linear.  The  length,  width,  and  length-to-width  ratio 
measurements  give  an  estimate  of  how  linear  an  indication  is.  but  may  not  always  cai)ture 
the  essential  information.  One  problem  is  that  the  pixels  are  arranged  in  an  ordered  two 
dimensional  array  which  imposes  some  restrictions  on  the  above  length  and  width  measure¬ 
ments.  The  measurements  will  always  be  along  the  primary  axes  of  the  array.  If  a  crack 
indication  aligns  with  one  of  the  axes,  then  the  distance  measures  and  their  ratio  will  give 
an  accurate  description  of  the  linearity  of  the  crack.  Cracks  can  form  in  any  direction  and 
if  a  crack  is  rotated  relative  to  the  array,  then  the  distance  measures  may  fail  to  capture 
the  crack’s  linearity.  Figure  G.2  displays  an  example  of  how  crack  orientation  influences 
the  length  and  width  measurements. 

Width  Width 


Figure  G.2.  Example  of  how  crack  orientation  relative  to  pixel  array  influences  the  length 
and  width  measurements.  The  measurements  for  the  crack  that  is  aligned 
with  the  array,  (a),  characterize  its  linearity.  The  second  crack  is  rotated 
relative  to  the  pixel  array,  (b),  and  the  length  is  similar  to  the  width,  even 
though  the  crack  is  piece-wise  linear. 


To  better  describe  linearity  the  software  should  measure  length  and  width  along  the 
orientation  of  the  crack.  Another  approach  to  determine  linearity  is  to  use  the  Hough 
transform.  The  Hough  transform  maps  from  an  [a-,  y]  Cartesian  coordinate  space  to  a 
parameterized  coordinate  space.  The  two  parameters  in  the  Hough  space  are  p  and  0, 
which  fully  describe  any  line  in  the  Cartesian  space.  The  parameter  p  is  the  length  of  a 
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line  segment  that  connects  the  origin  to  the  line  and  is  perpendicular  to  the  line.  The 
parameter  0  is  the  angle  the  line  segment  makes  with  the  x-axis.  Figure  G.3  shows  the 
relationship  between  the  Cartesian  and  Hough  parameters. 


Figure  G.3.  Relationship  between  parameters,  [a;,  2/],  and  the  Hough  transform  parame¬ 
ters,  [p^d]. 

The  Hough  coordinate  system  uses  the  two  parameters  to  represent  an  entire  line  in 
the  Cartesian  coordinate  space  as  a  single  point.  The  single  point  in  the  Hough  space  is 
actually  a  local  maximum  at  the  coordinates  [p,  9],  which  correspond  to  the  Cartesian  line. 
The  transformation  can  be  used  to  identify  linear  regions,  regardless  of  their  orientation 
relative  to  the  original  axes.  For  these  reasons,  the  Hough  transform  is  an  efficient  tool  to 
use  to  search  for  linear  FP  indications. 

The  Hough  transform  can  be  computed  in  two  ways:  the  accumulator  method  and 
the  Fourier  based  approach.  Both  approaches  provide  the  same  results,  so  the  accumu¬ 
lator  method  was  selected,  since  it  was  computationally  more  efficient  to  program  for 
this  application.  Equation  G.l  shows  the  fundamental  relationship  between  the  Cartesian 
coordinate  system  and  the  Hough  coordinate  frame. 

p  =  3-cos(^) -H  j/sin(0)  (G.l) 

In  applying  the  accumulator  method  to  calculate  the  Hough  transform,  each  point 
in  the  Cartesian  space  is  mapped  to  a  sinusoidal  curve  in  the  Hough  space.  The  sinusoidal 
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curve  is  all  points  in  the  Hough  space  that  satisfy  equation  G.l  for  a  given  [x,  y]  point.  To 
implement  the  transform,  the  Hough  coordinate  space  is  represented  by  a  two-dimensional 
array  with  one  axis  for  each  parameter.  The  p  axis  ranges  from  0  to  the  maximum  possible 
distance  in  the  original  image  (the  length  of  the  diagonal)  and  the  0  axis  ranges  from  0 
to  359  degrees.  For  a  given  [x,  y]  point,  the  transform  adds  the  value  one  to  the  Hough 
array  at  each  location  (cell)  which  approximates  the  actual  sinusoidal  curve.  The  original 
image  is  processed  pixel  by  pixel,  which  causes  the  Hough  array  to  accumulate  the  proper 
transform  values  in  each  cell.  The  Hough  cell  that  contains  the  largest  value  corresponds 
to  the  [/9,  0]  parameters  which  represent  the  Cartesian  line. 

Digital  image  processing  textbooks  contain  more  information  about  the  Hough  trans¬ 
form  (22,  44).  Hill’s  thesis  provides  several  examples  as  well  as  a  description  of  the  Fourier 
based  approach  to  calculating  the  transform  (26).  In  addition.  Crimson  et  al  describe  some 
of  the  limitations  of  the  Hough  transform  when  applied  to  noisy  images  (23). 

If  the  original  image  is  a  simple  line,  the  Hough  transforms  provides  an  efficient 
method  to  describe  the  location  and  orientation  of  the  line  relative  to  the  origin.  Crack 
indications  are  rarely  simple  lines  because  of  the  way  FP  bleeds  out  of  the  crack.  The 
indications  will  be  larger  than  a  single  pixel  thick  and  may  not  develop  in  a  straight  line. 
Since  the  images  are  more  complex  than  simple  lines,  the  results  of  the  Hough  transform 
must  be  interpreted  to  gain  insight  into  the  original  image. 

The  design  team  made  three  measurements  of  the  Hough  transform  to  characterize 
the  original  image.  The  first  measurement  was  the  maximum  value  in  the  Hough  space. 
The  maximum  value  corresponds  to  the  portion  of  the  original  indication  that  had  the 
largest  number  of  pixels  in  a  row.  A  relatively  large  ma.ximum  value  indicates  that  the 
original  image  contained  pixels  that  could  form  a  line.  A  noisy  image  that  is  not  linear 
could  also  create  a  large  value  in  the  Hough  space,  so  this  single  measurement  is  not 
sufficient. 

The  other  two  measurements  from  the  Hough  space  were  the  average  value  in  the 
array  and  the  variance  of  the  Hough  values.  Together,  these  two  measurements  describe 
how  the  Hough  values  are  distributed  and  should  help  characterize  the  linearity  of  iiidica- 
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tions.  For  example,  an  indication  that  is  strictly  linear  will  have  one  local  niaximuni  in 
the  Hough  space  along  with  many  values  that  are  much  smaller.  The  smaller  values  arise 
because  of  the  way  the  sinusoidal  curves  are  accumulated.  The  Hough  transform  of  a  linear 
indication  should  have  a  smaller  average  value  and  a  larger  variance  than  the  transform 
of  a  non-linear  indication.  These  three  Hough  features  will  be  collected  and  the  neural 
network  will  be  used  to  determine  if  they  increase  the  ability  to  differentiate  between  good 
and  defective  blades. 

G.3.2.4  Gradient  features.  In  addition  to  linearity,  another  important  feature 
that  human  inspectors  use  is  how  the  intensity  of  the  indication  changes  across  the  surface 
of  the  blade.  The  gradient  of  an  image  measures  intensity  changes,  so  it  is  the  logical 
choice  to  capture  additional  information  about  indications.  The  image  processing  software 
used  the  Sobel  operator  to  calculate  the  gradient,  which  is  a  standard  approach  described 
in  many  image  processing  texts  (22,  44).  There  are  other  similar  methods  to  calculate 
the  gradient,  including  Roberts’  operator,  as  well  as  Fourier  techniques,  which  can  use 
information  in  the  entire  image  to  calculate  the  gradient.  For  an  alternative  approach  to 
calculating  the  image  gradient,  refer  to  Tong’s  thesis  (60). 

The  Sobel  gradient  calculation  involves  applying  two  convolutions  to  the  original 
image.  The  convolutions  use  the  masks  Mr  and  My,  shown  below,  to  calculate  the  gradient 
in  the  x  and  y  directions. 
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To  perform  the  convolution,  the  masks  are  centered  at  a  location  in  the  array  and 
then  the  corresponding  elements  in  the  array  and  the  mask  are  multiplied  together  and 
summed.  Each  pixel  in  the  new  matrix  contains  the  result  of  the  convolution  when  the 
mask  was  centered  on  the  pixel.  The  two  convolution  masks  generate  two  matrices,  Gr  and 
Gy,  which  represent  the  image  gradient  in  each  direction.  The  two  directional  gradients 
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are  combined  element  by  element  using  equation  G.2.  The  resulting  values  in  matrix  Cl 
are  the  magnitudes  of  the  gradient  of  the  original  image. 

G=1g’+CJ)’  (G.2) 

Once  the  gradient  of  the  image  was  calculated,  the  image  processing  software  mea¬ 
sured  two  gradient  features  for  each  indication.  The  features  were  the  maximum  gradient 
and  the  average  gradient  within  the  indication.  Crack  indications  typically  have  intensity 
values  that  change  slowly  across  the  surface  of  the  blade  because  of  the  way  the  penetrant 
flows  out  of  the  crack.  Non-crack  indications  have  very  sharp  changes  in  intensity,  which 
corresponds  to  large  values  for  the  gradient.  The  two  gradient  measurements  do  not  rep¬ 
resent  all  of  the  gradient  information  in  each  indication,  but  they  are  relatively  easy  to 
collect  and  should  help  distinguish  between  crack  and  non-crack  indications. 

G.S.2.5  Summary  of  feature  extraction  process.  All  of  the  image  processing 
and  feature  extraction  software  was  developed  using  the  C  programming  language.  In 
the  overall  process,  the  actual  software  is  the  program  image. c,  shown  in  the  third  block 
in  figure  G.l.  In  addition,  figure  G.4  provides  a  more  detailed  description  of  the  image 
processing  and  feature  extraction  program. 

The  program  generates  features  with  a  label  of  0  for  non-crack  indications  and  a 
label  of  1  for  cracks.  The  following  list  summarizes  the  features  collected  by  the  program 
for  each  indication. 

Summary  of  features: 

1.  number  of  pixels 

2.  length  of  the  smallest  rectangle  that  bounds  the  indication 

3.  width  of  the  smallest  rectangle  that  bounds  the  indication 

4.  length-to-width  ratio 

5.  maximum  pixel  intensity 

6.  average  pixel  intensity 

7.  maximum  value  in  the  Hough  data  array 

8.  average  value  in  the  Hough  data  array 

9.  variance  of  the  values  in  the  Hough  data  array 
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Figure  G.4.  Image  processing  program  flowchart. 
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10.  maximum  gradient 

11.  average  gradient 

The  program  processed  the  140  swaths  of  data  and  identified  642  individual  indica¬ 
tions.  The  software  was  also  able  to  accurately  identify  and  label  the  183  crack  indications 
based  on  the  position  information  from  the  MathCAD  routine.  The  image  processing  por¬ 
tion  of  the  software  is  now  complete  and  the  labeled  indication  features  now  serve  as  the 
input  for  the  neural  network  portion  of  the  classification  software. 

G.4  Neural  Network  Classification 

With  a  data  set  of  features  established,  the  neural  network  can  now  be  trained  to 
classify  indications  into  cracks  and  non-cracks.  Since  the  objective  of  developing  the  soft¬ 
ware  is  to  demonstrate  the  concept  of  improving  the  classification  capability,  the  network 
was  kept  simple.  In  addition,  the  data  set  was  relatively  small,  which  limited  the  flexi¬ 
bility  in  terms  of  the  size  of  the  network  that  could  be  developed.  Developing  the  neural 
network  was  the  first  step  in  the  classification  portion  of  the  software.  Figure  G.5  displays 
the  second  row  of  figure  G.l  and  indicates  where  the  neural  network  fits  into  the  overall 
process. 


OwcripUon  of  Ortginal  data 


feature  vectora 


Figure  G.5.  Classification  portion  of  software  development. 


The  642  feature  vectors  collected  with  the  image  processing  routines  were  randomly 
shuffled  and  split  into  two  sets.  The  first  set  contained  two-thirds  of  the  data  (415  vectors) 
and  was  used  to  train  the  neural  network.  The  other  set  contained  the  remaining  227 
vectors  and  was  used  to  evaluate  the  performance  of  the  neural  network. 


G-15 


G.4-i  Network  Training.  The  design  team  developed  a  multilayer  perceplron  ( MLP ) 
using  the  LNKnet  software  from  MIT  Lincoln  Laboratory.  The  LNKnet  software  served 
as  a  tool  to  develop,  train,  and  test  a  variety  of  neural  network  structures.  To  execute,  the 
software  required  data  files  for  training  and  evaluation,  as  well  as  parameters  describing 
the  data  and  the  type  of  network  to  be  created.  The  software  then  used  the  information 
to  train  a  neural  network  and  report  on  the  error  rate  associated  with  the  network.  The 
software  used  a  standard  backpropagation  learning  rule  to  update  the  network  weights. 
When  training  was  complete,  the  software  provided  the  final  network  weights,  which  could 
be  used  to  implement  the  network.  There  are  a  variety  of  pattern  recognition  texts  that 
describe  in  detail  the  principles  behind  multilayer  perceptrons  as  well  as  other  neural  net¬ 
works  (47,  61) 

A  MLP  was  developed  with  an  input  layer,  one  hidden  layer,  and  an  output  layer  to 
process  the  indication  features.  Each  node  in  the  hidden  layer  received  as  input  the  dot 
product  of  the  appropriate  weight  vector  with  the  input  features.  The  output  of  the  hidden 
layer  used  the  standard  sigmoid  function.  Equation  G.3  shows  the  relationship  between 
the  input  feature  vector  and  output  of  the  hidden  layer  nodes. 

In  equation  G.3,  j/;(z)  is  the  output  of  the  jth  hidden  node,  x  is  the  input  feature 
vector,  and  wj"  is  the  transpose  of  the  weight  vector  associated  with  the  jfth  hidden  node. 
The  relationship  between  the  output  of  the  hidden  layer  and  the  output  of  the  output  layer 
is  the  same  as  the  relationship  between  the  input  layer  and  the  hidden  layer.  The  weight 
vectors  are  calculated  while  training  the  network  with  the  LNKnet  software. 

The  first  neural  network  used  all  11  features  for  the  input.  The  LNKnet. software 
was  initialized  to  create  22  hidden  nodes  and  2  output  nodes.  The  software  automatically 
added  one  node  to  each  layer,  resulting  in  a  network  with  12  input  nodes,  23  hidden  nodes, 
and  3  decision  nodes.  On  the  input  layer,  the  value  of  the  additional  node  was  set  equal 
to  one.  The  additional  hidden  and  output  layer  nodes  were  computed  in  the  same  manner 
as  all  of  the  other  nodes.  Only  two  of  the  output  nodes  were  used  to  classify  the  data  (see 
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section  G.5  for  a  complete  discussion).  Tlie  network  trained  for  500  epochs  and  achieved 
a  finaJ  error  rate  of  1.4%  with  the  training  data.  Figure  G.6  shows  the  neural  network's 
error  rate  as  it  trains. 


) 


The  fully  trained  network  had  an  error  of  6.6%  with  the  evaluation  data  set,  which  is 
reasonably  close  to  the  training  set  error  rate.  This  first  neural  network  training  example 
demonstrated  that  a  simple  multilayer  perceptron  could  learn  to  differentiate  between 
crack  and  non-crack  FP  indications.  Therefore,  the  features  collected  during  the  image 
processing  are  adequate  for  this  application. 

G.4-2  Analysis  of  Trained  Network.  To  verify  the  performance  of  the  neural  net¬ 
work,  the  data  vectors  were  randomly  shuffled  several  times  and  presented  to  the  LNKnet 
software  for  training.  The  accuracy  of  the  MLP  network  will  depend  on  the  specific  data 
sets  used  for  training  and  evaluation.  The  LNKnet  software  presented  the  data  vectors 
in  a  random  order,  but  the  shuffling  operation  mixed  the  data  vectors  between  the  train¬ 
ing  and  evaluation  sets.  The  results  of  training  a  network  with  the  new  data  sets  were 
consistent  with  the  first  neural  network.  For  example,  in  one  case  the  training  error  rate 
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Figure  G.6.  Neural  network  error  rate  while  training. 
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decreased  to  1.0%  and  the  evaluation  error  rate  increased  to  9.7%.  The  other  trials  had 
similar  results,  but  the  first  example  had  the  lowest  evaluation  error  rate,  so  that  network 
was  used  for  the  remainder  of  the  analysis. 

Another  method  to  check  the  performance  of  a  neural  network  is  to  compare  its 
results  to  other  traditional  classification  techniques.  The  LNKnet  software  provides  algo¬ 
rithms  to  implement  a  variety  of  decision  functions,  including  k-nearest  neighbor  (KNN) 
and  radial  basis  functions  (RBF).  Table  G.l  displays  the  results  of  applying  the  other  two 
algorithms  to  the  original  training  and  evaluation  data  sets. 


Algorithm 

Error  Rate  (%) 

KNN  k  =  2 

9.40 

k  =  3 

7.23 

k  =  4 

8.19 

k  =  5 

8.19 

k  =  6 

8.43 

RBF 

7.05 

Table  G.l.  Error  rates  for  alternate  classification  algorithms 


The  alternate  decision  techniques  for  classifying  the  data  had  similar  results  to  the 
MLP  network.  The  similarity  demonstrates  that  the  network  is  performing  at  least  as  well 
as  the  other  approaches  and  that  there  are  no  blatant  problems  with  the  network.  The 
MLP  requires  a  substantial  amount  of  processing  time  to  train  when  compared  to  the  KNN 
and  RBF  methods.  However,  the  neural  network  approach  can  offer  significant  advantages 
in  terms  of  execution  speed  and  memory  requirements  once  it  is  successfully  trained.  For 
a  thorough  review  of  the  advantages  and  disadvantages  of  different  classification  schemes, 
refer  to  Lippmann’s  articles  (33,  34). 

In  addition  to  the  above  empirical  tests,  there  are  theoretical  measures  used  to  gauge 
the  performance  of  a  neural  network.  The  ability  of  a  network  to  successfully  classify  a  set 
of  data  depends  on  the  size  of  the  training  set,  the  number  of  features,  and  the  number  of 
network  nodes.  There  are  no  exact  rules  that  dictate  how  large  a  network  should  be  for  a 
given  data  set  (in  the  above  example,  the  number  of  hidden  nodes  was  twice  the  number 
of  input  features),  but  there  are  three  guidelines  which  help  characterize  the  performance 
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of  a  network.  The  guidelines  can  also  suggest  when  a  network  structure  should  be  altered 
to  account  for  the  size  of  the  data  set  and  number  of  classification  categories. 

As  the  first  guideline,  Cover’s  rule  describes  the  network's  capacity  to  separate  the 
data  into  the  proper  classes  based  on  the  number  of  degrees  of  freedom  or  adjustable 
parameters  in  the  decision  function.  The  rule  is  also  referred  to  as  the  dichotomization 
capacity  (61).  For  data  that  is  well  distributed,  as  the  number  of  degrees  of  freedom 
increases,  the  probability  of  correctly  classifying  the  data  also  increases.  In  the  MLP 
network,  the  number  of  input  features  and  hidden  layer  nodes  determine  the  number  of 
degrees  of  freedom  in  the  decision  function.  Since  the  neural  network  used  11  inputs  and  23 
hidden  nodes,  it  has  a  higher  probability  of  correctly  classifying  the  FP  indications  when 
compared  to  the  FPIM’s  rule  based  decision  function  using  just  two  features. 

The  second  guideline  in  establishing  a  neural  network  is  Foley’s  rule.  The  rule  states 
that  the  number  of  training  vectors  per  class  should  be  at  least  three  times  the  number  of 
features  in  order  to  properly  train  the  neural  network.  The  above  example  has  11  features 
and  a  total  of  415  training  vectors.  In  the  training  set,  there  were  297  feature  vectors  of 
good  indications  and  118  feature  vectors  associated  with  defective  indications.  The  data 
satisfies  Foley’s  rule,  which  means  the  neural  network  is  not  just  memorizing  the  data  in 
the  training  set.  The  above  performance  in  classifying  the  evaluation  data  demonstrated 
the  network’s  ability  to  generalize  to  new  data. 

The  final  rule-of-thumb  is  Uncle  Bernie’s  rule,  developed  by  Dr  Bernard  Widrow, 
which  estimates  the  network’s  error  rate  based  on  the  number  of  training  vectors  and  the 
number  of  weights.  The  expected  error  rate  for  a  network  is  equal  to  the  number  of  weights 
in  the  network  divided  by  the  number  of  training  vectors.  For  example,  to  achieve  an  error 
rate  of  0.10,  the  network  should  be  trained  with  10  times  as  many  training  vectors  as 
there  are  adjustable  weights  in  the  network.  For  the  blade  classification  network,  there 
are  a  total  of  345  weights  (276  between  the  input  layer  and  hidden  layer  and  69  between 
the  hidden  layer  and  output  layer).  The  network  was  trained  with  only  415  vectors,  so 
according  to  Uncle  Bernie’s  rule,  it  would  be  expected  that  the  error  on  the  evaluation 
data  set  would  be  relatively  high  (83%).  Since  the  error  on  the  evaluation  vectors  was 
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significantly  smaller  than  expected,  6.651.  it  can  be  concluded  that  the  evaluation  data  is 
very  similar  to  the  training  data. 

Uncle  Bernie’s  rule  emphasizes  a  serious  deficiency  in  the  present  data  set:  it  is  too 
small.  The  network  performs  reasonably  well,  but  the  data  set  must  be  expanded  to  develop 
a  more  robust  decision  function.  See  section  G.6  for  a  complete  list  of  recommendations 
about  how  to  improve  the  classification  software. 

With  the  development  of  a  simple  neural  network,  the  majority  of  the  image  process¬ 
ing  and  cla.ssification  software  is  now  complete.  All  that  remains  is  to  interpret  the  output 
of  the  neural  network  in  a  manner  that  allows  SA-ALC  to  classify  blades  cost  efficiently. 

G.5  Neural  Network  Performance 

The  network  update  rules  used  in  section  G.4  treated  both  types  of  mistakes  equally 
when  training  the  neural  network.  To  apply  the  neural  network  to  the  blade  inspection 
problem,  the  output  of  the  network  must  be  interpreted  in  a  slightly  different  manner 
to  account  for  the  cost  of  the  different  types  of  classification  mistakes.  In  addition,  the 
network  results  should  be  adjusted  to  reflect  the  number  of  errors  made  per  blade  instead 
of  per  indication.  The  network  trained  above  used  indications  as  the  basic  unit  of  input 
and  there  are  several  indications  per  blade,  which  must  be  accounted  for  to  estimate  the 
probability  of  error  for  a  given  blade. 

G.5.1  An  Alternate  Interpretation  of  the  Network  Output.  The  neural  network  was 
trained  to  minimize  the  probability  of  error,  which  is  simply  misclassifying  an  input  vector. 
Since  there  are  two  possible  classes,  the  network  could  make  two  types  of  mistakes:  labeling 
a  good  blade  as  defective  or  labeling  a  defective  blade  as  good.  These  types  of  mistakes 
are  called  false  calls  and  missed  detections,  respectively.  The  costs  associated  with  the 
two  types  of  error  are  drastically  different  and  should  be  accounted  for  in  the  classification 
program.  A  false  call  removes  a  good  blade  from  service  or  may  cause  a  human  inspector 
to  spend  additional  time  checking  a  batch  of  blades.  A  missed  detection  allows  a  defective 
blade  to  return  to  service,  which  could  result  in  engine  failure,  loss  of  the  aircraft,  or  loss 
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of  life.  Clearly,  SA-ALC  requires  a  method  to  interpret  the  results  of  the  neural  network 
to  reflect  the  costs  associated  with  each  type  of  error. 

The  network  used  the  first  two  output  nodes  to  classify  each  input  vector.  The  output 
of  decision  node  j,  wdth  j  =  0, 1,  estimates  the  probability  of  the  input  vector  occurring 
if  the  vector  was  from  class  j.  In  other  words,  the  output  of  each  node  approximates 
the  conditional  probability,  P(.rpj),  where  f  is  the  input  vector  and  ujj  is  the  jth  class, 
with  j  =  0, 1.  The  network  assigns  the  label  to  the  input  that  corresponds  lo  the  highest 
probability  at  the  output.  The  results  of  this  type  of  network  are  interpreted  by  taking 
the  ratio  of  the  two  outputs  (node  0  output  divided  by  node  1  output)  and  comparing  the 
value  to  one.  If  the  ratio  is  greater  than  one,  then  the  input  is  labeled  good,  because  the 
probability  of  the  input  vector  belonging  to  class  tuo  is  higher  than  the  probability  of  the 
vector  belonging  to  class  Wi,  [P(x|u;o)  >  P(x|u;i)].  A  ratio  of  less  than  one  means  the 
input  should  receive  a  defective  label.  The  decision  boundary  occurs  at  the  point  where 
probabilities  are  equ«il. 

Comparing  the  ratio  of  the  conditional  probabilities  to  a  threshold  value  of  one 
assumes  the  costs  associated  with  each  type  of  error  are  the  same.  If  the  costs  are  unequal, 
the  ratio  of  the  probabilities  should  be  compared  to  a  different  threshold  value.  The  Bayes 
decision  criteria  allows  a  cost  to  be  assigned  to  every  outcome  of  a  binary  decision  problem. 
These  values  can  then  be  used  to  compute  the  decision  threshold  that  accounts  for  the 
costs  of  each  type  of  outcome.  (See  the  texts  by  Melsa  and  Cohn  or  Tou  and  Gonzalez 
for  a  complete  derivation  of  the  Bayesian  decision  criteria  (38,  61).)  For  example,  correct 
decisions,  such  as  labeling  a  non-cracked  blade  as  good  or  a  cracked  blade  as  defective, 
usually  have  a  cost  of  0.  On  the  other  hand,  decisions  which  represent  errors  typically  have 
positive  costs  and  the  magnitudes  reflect  the  relative  impact  of  each  type  of  error.  For  the 
blade  inspection  task,  SA-ALC  wants  to  minimize  the  number  of  missed  detections  and  is 
willing  to  accept  a  small  number  of  false  calls,  because  missed  detection  errors  would  have 
a  higher  cost  than  false  calls. 

The  exact  value  of  the  threshold  can  be  calculated  if  all  of  the  costs  and  the  prob¬ 
abilities  of  each  class  occurring  are  known  in  advance.  These  costs  and  probabilities  are 
difficult  to  quantify  for  the  blade  inspection  task.  To  demonstrate  the  concept  behind 
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accounting  for  the  different  types  of  error,  the  threshold  will  be  varied  across  a  range  of 
values  to  show  the  range  of  possible  error  rates. 

G.5.2  Classification  Error  Rates.  To  be  able  to  apply  a  variety  of  different  thresh¬ 
olds  to  the  outputs  of  the  network,  all  of  the  vectors  had  to  be  processed  through  the 
network  generated  in  section  G.4.  The  output  of  the  LNKnet  software  provided  all  of 
the  necessary  parameters  to  implement  the  network.  The  network  was  created  using  a  C 
program,  which  determined  the  values  of  the  output  layer  nodes  for  each  vector  in  the  data 
set.  The  second  block  in  figure  G.5  shows  where  this  program,  classify. c,  fits  into  the 
overall  classification  process. 

The  final  piece  of  software,  program  error_graphs.c,  used  the  output  values  gen¬ 
erated  by  classify. c  and  the  original  classification  labels  to  determine  error  rates  for 
different  threshold  values.  The  threshold  value  started  at  10“'^  and  increased  by  factors 
of  10  up  to  a  value  of  10®.  A  small  value  for  the  threshold  corresponds  to  a  relatively  smcdl 
cost  for  a  missed  detection  error  and  a  large  cost  for  a  false  call  error.  A  large  threshold 
value  implies  a  missed  detection  has  a  higher  cost  than  a  false  call.  .4s  the  threshold 
increases,  the  number  of  missed  detection  errors  will  decrease  and  the  number  of  false 
calls  will  increase.  For  example,  a  threshold  value  of  10  means  that  the  output  of  node  0 
(good  indication)  must  be  more  than  10  times  larger  than  the  output  of  node  1  (defective 
indication)  for  the  program  to  label  the  input  vector  as  a  good  indication.  The  results  of 
applying  different  decision  thresholds  are  shown  in  figure  G.7. 

The  two  lower  curves  in  figure  G.7,  with  small  circles  at  each  data  point,  represent 
the  false  call  and  missed  detection  error  rates  for  the  network  when  all  642  features  vectors 
were  used  for  input.  Since  each  feature  vector  represents  one  indication,  the  lower  curves 
are  error  rates  per  indication  and  not  for  an  entire  blade.  The  data  had  approximately  one 
crack  indication  per  blade,  which  means  the  curve  for  missed  detections  also  accurately 
represents  the  missed  detection  error  rate  per  blade. 

The  blades  had  an  average  of  10  noii-crack  indications  per  blade.  The  number  of 
non-crack  indications  per  blade  considers  the  amount  of  overlap  between  the  swaths.  For 
the  ninth  stage  blades  used  to  gen  uat  j  the  data  set,  the  swaths  overlap  substantially,  so  10 


Figure  G.7.  False  call  and  missed  detection  errors  for  different  decision  threshold  values. 

indications  is  a  conservative  estimate.  The  false  call  error  rate  per  blade  can  be  calculated 
using  equation  G.4,  if  the  indications  independently  influence  the  overall  error  rate. 


^Jc/blade  —  1  (l  Ejc/indication) 


(G.4) 


In  equation  G.4,  Ejdbiade  is  the  false  call  error  rate  per  blade,  Ejdindication  is  the 
false  call  error  rate  per  indication,  and  n  is  the  number  of  indications  per  blade.  This 
relationship  was  used  to  generate  the  curve  for  false  call  rate  per  blade  in  figure  G.7,  with 
small  squares  at  each  data  point.  The  false  call  error  rate  per  blade  is  an  estimate  of  the 
probability  of  clzissifying  a  blade  with  10  non-crack  indications  as  defective.  This  error 
rate  can  be  used  to  characterize  the  system’s  performance  while  processing  good  blades. 
As  shown  in  the  figure,  the  error  rate  per  blade  can  approach  100%  as  the  false  call  rate 
per  indication  increases. 
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The  information  contained  in  the  graph  allows  SA-ALC  to  select  the  decision  thresh¬ 
old  which  provides  the  most  cost  effective  error  rates  for  the  blade  inspection  task.  For 
example,  at  a  threshold  of  1,  the  missed  detection  error  rate  is  5.5%  and  the  false  call 
error  rate  per  blade  is  21.5%).  Increasing  the  threshold  to  10  decreases  the  missed  de¬ 
tection  error  to  4.4%i  and  increases  the  false  call  error  rate  to  41.6%.  The  FPIM  has  a 
false  call  error  rate  of  at  least  50%  and  an  unknown  missed  detection  error  rate  ( SA-ALC 
does  not  measure  the  missed  detection  error  rates  for  their  system,  but  it  is  probably  rela¬ 
tively  small — approximately  1%.)  The  decision  function  using  the  neural  network  output 
offers  improved  false  call  error  rates  and  more  flexibility  in  selecting  how  the  automated 
inspection  system  operates. 

Another  way  to  analyze  the  performance  of  the  above  neural  network  is  to  compare  its 
results  to  a  neural  network  developed  using  only  the  features  processed  by  the  FPIM.  The 
FPIM  uses  a  complex,  rule-based  classification  routine,  which  utilizes  only  two  features, 
number  of  pixels  and  maximum  pixel  intensity,  to  categorize  blades.  The  small  feature  set 
limits  the  FPIM’s  ability  to  differentiate  between  good  and  defective  blades.  The  MLP 
network  developed  above  had  11  input  features  and  23  hidden  nodes,  which  combined  to 
provide  many  degrees  of  freedom  and  increase  the  probability  of  classifying  the  data  into 
the  proper  categories.  Simulating  the  entire  FPIM  software  is  beyond  the  scope  of  this 
project,  but  it  is  relatively  easy  to  develop  a  MLP  network  using  the  same  input  features 
as  the  FPIM.  A  neural  network  approximation  cannot  completely  characterize  the  FPIM 
software,  but  it  can  offer  a  reasonable  estimate  which  demonstrates  the  advantages  of  a 
network  with  more  degrees  of  freedom. 

The  second  MLP  network  was  developed  with  the  same  process  used  to  create  the 
first  neural  network.  The  input  feature  set  was  limited  to  contain  just  indication  size  and 
maximum  pixel  intensity.  The  network  had  an  input  layer,  one  hidden  layer  with  five 
nodes,  and  an  output  layer  with  three  nodes.  The  LNKnet  software  received  the  same 
combination  of  feature  vectors  used  with  the  first  network  to  train  the  new  network.  The 
network  achieved  a  final  error  rate  of  5.3%i  with  the  training  data  and  an  error  rate  of  6.17% 
with  the  evaluation  data.  The  figure  G.8  displays  the  network  error  rate  as  a  function  of 
the  number  of  training  epochs. 
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Figure  G.8.  Neural  network  error  rate  while  training. 


The  training  error  rate  curve  for  the  network  developed  with  two  input  features  shown 
in  figure  G.8  is  fundamentally  different  than  the  training  error  for  the  network  with  all  11 
input  features  shown  in  figure  G.6.  The  second  network  did  not  demonstrate  significant 
learning  and  its  error  curve  did  not  converge  to  a  minimum  value.  Although  the  error 
curve  was  noisy,  the  second  network  was  still  able  to  correctly  classify  most  of  the  feature 
vectors.  The  error  rate  for  the  evaluation  data  was  slightly  better  than  the  corresponding 
error  rate  for  the  first  network,  but  the  final  error  rates  for  the  training  data  for  the  two 
network  were  noticeably  different.  Table  G.2  summarizes  the  results. 


Error  Rates  (%) 

Training  Data 

Evaluation  Data 

MLP  with  11  features  (11-23-3) 

1.4 

6.61 

MLP  with  2  features  (2-5-3) 

5.3 

6.17 

Table  G.2.  Error  rates  for  different  MLP  network  structures. 


To  complete  the  comparison  between  the  original  MLP  and  the  MLP  used  to  char¬ 
acterize  the  FPIM,  the  second  network  was  implemented  using  a  modified  version  of  the 
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program  classify. c.  The  output  of  the  classification  program  was  then  processed  using 
the  technique  discussed  above  to  generate  false  call  and  missed  detection  curves.  The  deci¬ 
sion  threshold  varied  over  the  same  range  as  for  the  previous  network.  Figure  G.9  displays 
the  error  rates  for  the  MLP  approximation  of  the  FPIM  decision  software. 


Error  Rates  for  2  Feature  Neural  Network  Classification 


10-6  JO-'*  10-2  10  102  10'*  10®  10® 

Threshold  Ratio 


Figure  G.9.  False  call  and  missed  detection  errors  for  MLP  approximation  of  FPIM  de¬ 
cision  software. 


As  expected,  the  neural  network  approximation  for  the  FPIM  decision  software  did 
not  perform  as  well  as  the  more  advanced  network  using  a  large  set  of  input  features  for 
each  indication.  Figure  G.9  shows  how  both  types  of  error  rates  change  rapidly  as  the 
threshold  value  changes.  When  the  threshold  was  reduced  from  one  by  three  orders  of 
magnitude,  the  decision  function  was  labeling  all  indications  as  good,  which  means  the 
network  misclassified  all  of  the  defective  blades.  Similarly,  when  the  threshold  increased 
three  orders  of  magnitude  from  one,  all  indications  were  labeled  defective  and  the  false  call 
rate  per  blade  increased  to  its  maximum  value. 
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The  new  decision  function  clearly  provides  less  flexibility  than  the  previous  network  in 
terms  of  how  the  output  of  the  neural  network  is  interpreted.  In  addition,  the  magnitudes 
of  the  missed  detection  and  false  call  errors  are  signiflcantly  larger  for  the  second  network 
than  in  the  original  network.  Table  G.3  compares  the  error  rates  of  the  two  networks  for 
three  different  threshold  values. 


Decision  Threshold 

Network  Structure 

Error  Rates  (%) 

False  Call 

Missed  Detection 

10-1 

MLP  11  features 

16.1 

6.0 

MLP  2  features 

4.3 

29.5 

O 

O 

MLP  11  features 

21.5 

5.5 

MLP  2  features 

40.2 

8.2 

10+1 

_ _ _ 1 

MLP  11  features 

41.6 

4.4 

MLP  2  features 

86.4 

1.6 

Table  G.3.  Comparison  of  false  call  per  blade  and  missed  detection  error  rates  for  different 
MLP  network  structures. 


The  MLP  with  11  input  features  has  neither  the  lowest  false  call  error  rate  nor  the 
lowest  missed  detection  error  rate  across  the  entire  range  of  thresholds  show'n  in  table  G.3. 
However,  the  overall  trend  between  the  two  MLP  structures  shows  that  the  11  feature 
network  is  more  consistent  and  has  more  gradual  changes  in  the  error  rates  as  the  threshold 
value  changes  when  compared  to  the  network  with  2  input  features. 

The  neural  network  approximation  of  the  FPIM  classification  software  is  a  very  rough 
estimate,  but  it  does  illustrate  some  of  the  problems  with  using  only  two  features  to  classify 
blades.  Two  input  features  severely  restrict  the  number  of  degrees  of  freedom  available  to 
the  network  and  limit  its  capacity  to  separate  the  two  classes.  Expanding  the  feature  set 
gives  the  network  more  flexibility  and  increases  the  probability  of  differentiating  between 
good  and  defective  blades. 


G.6  Summary  and  Recommendations 

The  advanced  image  processing  and  classification  software  successfully  demonstrated 
an  improved  classification  ability  compared  to  the  existing  FPIM  software.  The  improve- 
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merits  came  from  using  simple  image  processing  routines  to  collect  a  variety  of  features 
from  each  FP  indication.  The  new  features  were  selected  to  extract  information  that  of¬ 
fered  a  more  complete  description  of  crack  indications.  The  expanded  feature  set  allowed 
for  the  development  of  multilayer  perception  network  with  more  degrees  of  freedom  than 
the  FPIM’s  rule-based  decision  software.  The  output  of  the  neural  network  was  inter¬ 
preted  using  Bayes  decision  analysis  to  adjust  the  false  call  and  missed  detection  error 
rates.  The  final  network  was  compared  to  a  simple  MLP  network  developed  using  only 
the  features  used  by  the  FPIM.  The  comparison  served  only  as  an  illustrative  example  of 
how  the  expanded  feature  set  improved  the  system’s  classification  performance.  For  addi¬ 
tional  information,  see  appendix  H,  which  presents  an  alternative  method  for  evaluating 
the  performance  of  the  classification  software  in  terms  of  dollar  cost. 

The  next  generation  automated  blade  inspection  system  will  require  advanced  image 
processing  and  classification  software  in  order  to  achieve  acceptable  false  call  and  missed 
detection  error  rates.  The  software  developed  above  demonstrated  the  potential  perfor¬ 
mance  advantages  of  such  a  system,  but  it  is  only  a  starting  point.  Research  on  this  topic 
could  continue  on  a  number  of  paths. 

The  first  step  required  to  more  fully  develop  the  software  is  to  collect  better  data. 
As  discussed  in  section  G.2.2,  the  data  set  must  be  significantly  larger,  contain  a  variety 
of  different  types  of  blades,  and  have  both  defective  and  good  blades.  In  brief,  the  data 
must  be  representative  of  the  type  of  data  SA-ALC  would  expect  the  automated  blade 
inspection  system  to  process.  In  addition,  it  would  be  advantageous  to  receive  the  data  in 
a  raw  format  without  any  preprocessing  that  could  distort  the  results. 

If  a  new  data  set  is  developed,  it  should  be  labeled  directly  by  trained  inspectors 
who  have  access  to  the  original  blades  and  know  how  to  interpret  the  different  types  of 
indications.  The  inspectors  should  label  the  image  data  in  an  electronic  format  so  there  is 
no  ambiguity  and  to  help  reduce  the  probability  of  errors.  In  addition,  the  inspectors  could 
label  all  indication  types,  such  as  cracks,  scratches,  pits,  and  dents,  which  may  be  useful 
in  developing  more  advanced  software.  An  increased  number  of  classes  would  require  a 
corresponding  increase  in  the  number  of  feature  vectors  to  properly  train  the  network. 
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Another  improvement  in  the  data  set  would  be  to  collect  data  using  a  charge  coupled 
device  (CCD)  camera  instead  of  the  FPIM’s  sensor.  The  FPIM’s  scanning  laser  sensor  was 
the  only  available  source  of  image  data  for  this  study.  The  next  generation  sensor  will 
likely  use  a  CCD  camera  to  collect  indication  data,  so  the  same  type  of  data  should  be 
used  to  effectively  train  a  neural  network. 

Besides  improving  the  data  set,  future  research  could  explore  alternative  image  pro¬ 
cessing  methods  to  collect  different  types  of  features.  For  example,  there  are  a  variety 
of  transform  techniques,  including  Fourier  (42),  Gabor  (3,  59),  and  wavelet  based  trans¬ 
forms  (35),  that  can  efficiently  extract  information  from  images.  Another  potential  ap¬ 
proach  is  to  use  morphology  to  process  the  image  data  and  extract  information  (36.  57). 
Morphology  treats  gray-scale  images  as  three-dimensional  structures  and  uses  set  theory 
to  manipulate  the  images  and  collect  features.  The  above  analysis  used  the  Hough  trans¬ 
form  and  the  image  gradient  to  extract  a  few  simple  features.  These  techniques  could  be 
further  developed  to  use  the  Hough  and  gradient  information  more  completely,  instead  of 
measuring  a  few  values  from  each  transformed  image  (26,  60).  Other  indication  features, 
such  as  how  the  FP  changes  over  time  or  features  extracted  from  color  images,  should  also 
be  explored  to  determine  if  they  can  improve  the  performance  of  the  blade  classification 
software. 

Finally,  future  research  should  develop  alternative  classification  methods  by  examin¬ 
ing  other  neural  network  structures  and  decision  functions.  The  list  of  possible  variations 
is  extensive  (19,  32,  58).  At  a  minimum,  the  research  should  determine  the  saliency  of 
the  indication  features  to  select  the  most  productive  subset  and  increase  the  speed  of  the 
feature  extraction  process  (49).  Neural  networks  that  process  the  whole  image  (as  a  sin¬ 
gle  unit  or  a  collection  of  subregions,)  instead  of  extracting  features  from  each  indication, 
may  also  offer  advantages  worth  exploring.  In  addition,  a  fuzzy  logic  approach  to  clas¬ 
sify  the  indications  may  offer  an  efficient  method  to  represent  the  rules  used  by  human 
inspectors  (30,  43).  There  are  clearly  numerous  options  available  to  continue  the  technical 
research  in  this  area  to  improve  the  blade  inspection  process. 
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Appendix  H.  Evaluation  of  Classification  Software  in  Terms  of  Cost 


H.l  Introduction 

Software  was  presented  in  appendix  G  to  perform  image  processing  and  blade  classifi¬ 
cation  for  the  automated  inspection  systems  described  in  alternatives  2  and  3.  The  previous 
analysis  concentrated  on  the  technical  aspects  of  the  algorithms  and  demonstrated  that  a 
neural  network  approach  could  improve  the  inspection  system’s  classification  accuracy  by 
analyzing  more  features  from  each  indication.  The  following  analysis  quantifies  the  perfor¬ 
mance  of  the  classification  algorithms  in  terms  of  annual  cost  by  translating  the  error  rates 
into  monetary  values.  The  conversion  from  error  rates  to  dollars  was  performed  using  the 
model  described  in  appendix  D.  The  results  indicate  that  the  network  using  11  features 
was  able  to  achieve  a  marginally  lower  annual  cost  compared  to  the  network  using  2  input 
features. 

H.2  Cost  of  Different  Classification  Errors 

The  analysis  performed  on  the  image  processing  and  blade  classification  software  in 
appendix  G  concluded  with  two  graphs  describing  the  estimated  error  rates  for  two  clas¬ 
sification  algorithms.  Both  algorithms  used  simple  neural  network  structures  (multilayer 
perceptrons)  to  classify  fluorescent  penetrant  indications.  The  first  algorithm  used  11  fea¬ 
tures  to  classify  indications,  while  the  second  algorithm  relied  on  only  2  features.  The 
missed  detection  and  false  call  error  rates  achieved  with  each  algorithm  were  used  to  eval¬ 
uate  their  performance.  The  error  rates  were  also  used  as  supporting  evidence  for  the 
expected  performance  of  the  designs  for  alternatives  2  and  3. 

The  error  rates  for  each  type  of  algorithm  provide  one  measure  of  the  system’s  perfor¬ 
mance.  An  alternative  way  to  evaluate  the  algorithms  is  to  determine  the  expected  cost  of 
operating  an  inspection  system  with  the  given  error  rates.  The  error  rates  can  be  converted 
into  monetary  values  by  estimating  the  number  of  bad  blades  returned  to  service  by  the 
inspection  system.  The  number  of  bad  blades  produced  can  be  used  to  predict  the  number 
of  unscheduled  maintenance  events  and  the  associated  maintenance  cost.  In  addition,  the 
number  of  blades  replaced  unnecessarily  and  their  cost  can  also  be  estimated.  The  model 
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for  the  cost  of  incorrectly  classifying  blades  described  in  appendix  D  performs  exactly  this 
type  of  conversion  between  inspection  error  rates  and  expected  cost. 


The  error  rates  for  the  two  classification  networks  are  shown  in  figures  G.7  and  G.9. 
These  values  were  used  as  inputs  to  the  model  for  the  cost  of  incorrectly  classifying  blades 
to  determine  the  expected  cost  of  each  approach.  The  graph  shown  in  figure  H.l  displays 
the  results  of  the  conversion  from  error  rates  to  cost. 


Figure  H.l.  Projected  annual  cost  of  the  inspection  system  operating  with  different  error 
rates  for  the  classification  software.  The  costs  are  based  on  the  cost  of 


unscheduled  maintenance  events  and  the  cost  of  replacing  blades. 


The  curves  in  the  graph  show  that  the  cost  performance  of  the  two  claissification 
algorithms  is  quite  similar.  The  11  feature  network  offers  lower  or  equivalent  cost  than 
the  2  feature  network  at  all  but  3  of  the  threshold  values.  (The  threshold  values  describe 
how  the  output  of  the  neural  network  is  interpreted.  See  section  G.5.1  for  details.)  In 
addition,  the  11  feature  network  achieved  the  minimum  expected  cost  of  $91. 9M  per  year 
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at  a  threshold  of  10~^.  It  is  worthwhile  to  note,  however,  that  at  this  threshold  value, 
the  11  feature  network  achieved  a  missed  detection  rate  of  43  percent  and  a  false  call  rate 
of  0  percent.  These  error  rates  clearly  represent  unacceptable  inspection  performance. 
However,  the  minimum  cost  for  the  11  feature  network  was  only  $2.7M  lower  than  the 
minimum  cost  for  the  2  feature  network  (a  difference  of  3%).  The  costs  associated  with 
the  11  feature  network  are  significantly  lower  than  the  costs  for  the  2  feature  network  at 
threshold  values  of  1,  10,  and  10^.  However,  the  costs  at  these  three  threshold  values  are 
not  important  because  they  are  significantly  higher  than  the  global  minimum  cost  in  the 
graph.  Clearly,  the  11  feature  network  does  not  significantly  improve  the  classification 
accuracy  of  the  inspection  system  when  the  performance  is  measured  in  terms  of  cost. 

It  is  important  to  understand  that  the  costs  presented  in  figure  H.l  are  for  comparison 
purposes  only.  The  values  show  the  relative  performance  of  the  two  classification  schemes 
using  a  monetary  scale,  but  the  values  should  not  be  misinterpreted  as  the  estimated  cost 
of  the  proposed  inspection  system.  These  cost  values  are  not  used  in  the  evaluation  process. 

Realistically,  any  classification  algorithm  would  have  to  achieve  much  lower  missed 
detection  and  false  call  rates  in  order  to  be  implemented  as  a  viable  inspection  system. 
The  error  rates  specified  for  the  alternative  2  design,  1%  for  missed  detections  and  20% 
for  false  calls,  are  reasonable  goals  for  the  classification  algorithm.  With  the  proper  re¬ 
search,  the  neural  network  classification  approach  should  be  able  to  achieve  such  error 
rates  within  three  years.  As  mentioned  in  appendix  G,  the  primary  obstacle  to  improving 
the  classification  software  is  the  quality  and  quantity  of  the  data  used  to  train  the  network. 

The  neural  networks  developed  in  appendix  G  only  classify  the  blades  into  two  cate¬ 
gories  (good  and  suspect).  The  design  for  alternative  2  uses  three  classification  categories 
(good,  bad,  and  suspect)  which  can  further  improve  the  performance  of  the  inspection  sys¬ 
tem.  The  additional  improvements  arise  by  allowing  the  automated  portion  of  the  system 
to  classify  blades  as  bad,  which  reduces  the  probability  that  human  inspectors  will  receive 
bad  blades  to  inspect.  The  reduction  in  the  number  of  bad  blades  sent  to  the  human  in¬ 
spectors  gives  them  fewer  opportunities  to  make  mistakes.  Overall,  the  automated  system 
that  can  classify  blades  into  three  categories  effectively  reduces  the  cost  by  minimizing  the 
number  of  bad  blades  called  good. 
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The  classification  algorithm  using  neural  networks  offers  an  approach  that  may  im¬ 
prove  the  inspection  performance  of  the  current  system.  The  analysis  in  appendix  G 
demonstrated  that  the  approach  was  a  viable  option,  but  it  still  requires  extensive  re¬ 
search.  The  additional  research  must  fully  develop  and  characterize  the  performance  of 
the  neural  network  and  explain  how  it  will  meet  the  needs  of  the  Air  Force  for  jet  engine 
blade  inspection. 

H.3  Summary 

This  analysis  used  the  model  developed  for  the  cost  of  incorrectly  classifying  blades  to 
quantify  the  costs  associated  with  inspection  errors  described  in  the  results  of  appendix  G. 
The  error  rates  between  the  two  classification  algorithms  are  significantly  different  when 
viewed  in  their  original  form,  but  the  differences  fade  when  the  results  are  transformed 
into  the  expected  cost  of  each  system.  In  terms  of  cost,  the  classification  neural  network 
using  11  input  features  offered  slightly  better  performance  than  the  network  using  2  input 
features.  The  cost  values  indicate  that  the  error  rates  achieved  with  the  two  networks  are 
unacceptably  high  and  must  be  improved  in  order  for  the  system  to  become  operational. 
Additional  research  must  be  completed  to  bring  the  error  rates  down  to  reasonable  levels, 
such  as  1%  for  the  missed  detection  rate  and  20%  for  the  false  call  rate.  This  research 
could  be  completed  within  the  next  three  years.  The  transformation  from  classification 
error  rates  to  expected  cost  provides  an  alternative  way  to  view  the  performance  of  the 
neural  network  software.  The  conversion  does  not  influence  the  final  evaluation  of  the 
alternatives  or  the  recommendations  made  in  chapter  VI. 
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Appendix  I.  Human  Factors  Research  Related  to  Inspection 


The  Integrated  Blade  Inspection  System  (IBIS)  uses  human  inspectors  to  look  for 
cracks  in  engine  parts.  A  human  inspector’s  performance  at  detecting  a  signal  (any  relevant 
indication,  such  as  a  crack)  during  an  inspection  task  depends  on  a  variety  of  factors.  One 
important  factor  is  the  probability  of  a  signal  occurring.  For  the  IBIS  design  project, 
the  unoptimized  version  of  alternative  2,  (discussed  in  section  D.3.3.3)  should  increase 
the  concentration  of  bad  blades  in  the  batch  of  blades  presented  to  the  human  inspector 
compared  to  alternative  1  and  the  baseline  systems.  The  increase  in  the  concentration 
of  bad  blades  should  cause  an  increase  in  the  probability  of  the  inspector  identifying  the 
crack. 

Several  human  factors  experiments  have  quantified  the  effects  of  the  signal  probability 
on  inspector  performance  in  different  laboratory  and  industrial  environments.  The  primary 
results  by  Colquhoun  in  a  laboratory  setting  showed  that  an  increase  in  the  probability  of 
a  signal  will  increase  the  probability  of  an  inspector  detecting  the  signal  (11).  Colquhoun 
was  able  to  measure  a  significant  increase  in  detection  efficiency  for  a  sixfold  increase  in 
signal  probability. 

Another  study  by  Fox  and  Htislegrave  verified  the  results  of  Colquhoun’s  work  in  an 
industrial  inspection  task  (21).  Colquhoun’s  study  used  relatively  large  signal  probabili¬ 
ties  and  changes,  such  as  a  change  from  a  signal  probability  of  8%  to  50%.  For  a  typical 
inspection  task,  the  signal  probabilities  are  usually  much  lower.  For  example,  the  IBIS 
facility  expects  5%  of  the  blades  to  be  cracked.  Fox  and  Haslegrave  demonstrated  that 
inspector  performance  could  improve  significantly  in  a  real  world  environment,  even  for 
small  changes  in  the  signal  probability.  They  showed  that  the  inspector  detection  perfor¬ 
mance  improved  when  the  probability  of  a  signal  occurring  increased  from  1%  to  5%.  Their 
study  also  revealed  that  increases  in  probability  as  small  as  ten  percent  could  significantly 
enhance  an  inspector’s  detection  performance. 

Additional  experiments  have  confirmed  the  above  results  under  different  inspection 
conditions  (18,  66).  This  evidence  indicates  that  the  human  inspector’s  performance  should 
be  improved  by  increasing  the  probability  of  presenting  a  cracked  blade  for  inspection. 
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The  old  alternative  2  design  will  automatically  sort  blades  into  two  categories:  good 
and  suspect.  The  good  blades  will  exit  the  system  and  the  suspect  blades,  which  contain  a 
very  high  percentage  (99%)  of  the  cracked  blades,  will  be  presented  to  the  human  inspector. 
A  defective  blade  is  any  blade  with  a  relevant  indication,  including  cracks,  scratches,  pits, 
and  dents.  In  the  IBIS  facility,  the  inspectors  are  primarily  concerned  with  identifying 
cracks,  so  for  simplicity,  all  defective  blades  will  be  refered  to  as  cracked  blades.  Since  a 
portion  of  the  good  blades  were  removed,  the  probability  of  a  cracked  blade  being  in  the 
remaining  batch  of  suspect  blades  has  increased  compared  to  the  original  set  of  blades. 
For  example,  suppose  the  system  receives  a  set  of  blades  and  5%  of  those  blades  are 
actually  cracked.  If  the  automatic  inspection  system  determines  that  25%.  of  the  blades 
are  suspect  (including  all  that  are  actually  cracked),  then  the  probability  of  a  cracked  blade 
has  increased  from  5%  to  20%,  (^  =  20%).  With  this  percentage  of  cracked  blades,  the 
inspector  should  have  a  higher  probability  of  detecting  a  crack. 

Although  the  inspector’s  performance  should  improve,  it  is  difficult  to  quantify  the 
expected  changes  in  detection  efficiency  without  conducting  specific  experiments  on  a  par¬ 
ticular  task  (31).  For  the  old  alternative  2,  an  estimate  will  be  made  on  how  a  change  in 
signal  probability  will  influence  the  inspector’s  performance. 

To  quantify  the  effect  of  a  change  in  signal  probability,  first  assume  the  future  inspec¬ 
tion  system  receives  a  total  of  1.2  million  blades  per  year  that  can  be  processed  automati¬ 
cally.  If  5%  of  these  blades  are  actually  cracked,  then  there  are  a  total  of  60,000  defective 
blades  that  a  human  inspector  could  potentially  identify.  Although  the  inspectors  at  the 
IBIS  facility  are  well  trained,  they  will  occasionally  make  errors  and  allow  a  cracked  blade 
to  pass  through  without  detection.  If  the  inspectors  have  a  98%  accuracy  rate  of  detecting 
bad  blades  for  the  original  configuration,  then  they  miss  1200  cracked  blades  per  year. 
The  addition  of  an  automatic  inspection  station  will  increase  the  density  of  cracked  blades 
presented  to  the  inspectors  and  should  improve  iheir  accuracy  rate.  If  their  accuracy  in¬ 
creases  by  as  little  as  one  percent,  they  will  detect  an  additional  600  cracked  blades  per 
year. 

In  terms  of  the  decision  analysis  between  the  alternatives,  delecting  cracks  is  a  very 
important  criteria.  Even  though  600  blades  is  a  relatively  small  amount  compared  to 
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the  1.2  million  blades  processed  each  year,  it  has  measurable  value  in  the  analysis  in  terms 
of  improving  inspector  consistency. 

Design  alternative  1  had  a  detection  probability  of  0.90  for  the  human  inspectors, 
which  was  the  same  as  the  baseline.  As  mentioned  above,  the  exact  increase  in  detection 
probability  cannot  be  quantified  without  direct  experiments,  but  a  reasonable  estimate  can 
be  made.  For  this  study,  it  was  assumed  the  detection  probability  of  a  human  inspector 
increased  to  0.95  for  the  previous  version  of  alternative  2. 

The  assumption  about  the  impact  of  increasing  the  probability  of  cracked  blades  on 
human  inspector  performance  is  clearly  subjective.  A  more  accurate  analysis  would  be 
beyond  the  scope  of  this  study. 

In  addition  to  the  probability  of  a  signal  occurring,  there  are  a  variety  of  other  factors 
that  impact  an  inspector's  performance  (54).  These  factors  include  the  following: 

1.  The  duration  of  the  task  without  breaks 

2.  The  number  and  type  of  defects  to  search  for 

3.  The  signal  detectability 

4.  The  rate  of  presentation  of  samples  or  the  number  of  samples  inspected  per  shift 

5.  The  number  of  inspectors  in  series,  if  each  has  the  ability  to  individually  detect  the 

signal 

6  Whether  the  signal  is  stationary  or  moving 

The  alternatives  developed  in  the  study  may  affect  the  inspectoi  's  ability  to  detect 
cracks  with  respect  to  at  least  two  of  the  above  items.  First,  implementing  alternative  1  and 
using  statistical  quality  control  to  improve  the  performance  of  the  .4FPPM  may  improve 
the  detectability  of  indications  on  the  blades.  This  type  of  improvement  would  likely  have 
the  same  affect  on  all  three  aJternatives.  The  second  factor  is  how  alternative  2  may 
decrease  the  rate  of  blade  presentation  to  the  inspectors.  A  lower  presentation  rate  will 
improve  an  inspector’s  probability  of  detecting  an  indication  (66).  The  throughput  analysis 
assumes  that  under  alternative  2,  the  inspectors  will  have  to  examine  approximately  20% 
of  the  blades  that  they  inspect  under  alternative  1  (see  sections  4.2  and  5.3.1).  As  with 
signal  probability,  quantifying  the  difference  between  the  alternatives  with  respect  to  the 
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detectability  of  the  cracks  or  the  number  of  blades  presented  to  the  inspector  would  require 
subjective  estimates.  Fully  developing  the  estimates  would  require  additional  research 
beyond  the  scope  of  this  study. 


The  IBIS  facility  management  should  be  aware  of  all  of  the  above  factors  and  how 
they  influence  inspector  performance  to  optimize  the  probability  of  detecting  a  crack.  For 
additional  information,  the  compendium  compiled  by  Boff  and  Lincoln  is  an  excellent, 
current  reference  that  describes  human  perception  and  performance  over  an  extensive 
range  of  activities,  including  inspection  (6). 
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Appendix  J.  Sensor  Rtquirements 


J.l  Introduction 

The  sensor  subsystem  is  composed  of  a  charge  coupled  device  (CCD)  camera,  a  lens, 
and  a  light  source.  The  specifications  for  these  components  are  driven  by  the  requirements 
of  the  inspection  system  to  detect  crack-like  indications.  The  calculations  assume  the 
following  requirements: 

•  The  resolution  on  the  blade  surface  is  0.005" 

•  The  largest  part  to  be  inspected  is  10"  x  10"  x  4" 

•  The  fluorescent  penetrant  used  will  emit  light  with  a  wavelength  of  490nni 

•  The  fluorescent  penetrant  is  designed  to  respond  to  incident  light  with  a  wavelength 
of  365nm 

•  The  intensity  of  the  incident  light  at  the  surface  of  the  blade  must  be  at  least 
800pW/cm^ 

Using  these  requirements,  each  component's  critical  specifications  in  the  sensor  subsystem 
are  determined. 


J.2  Light  Source 

The  light  source  is  made  up  of  an  emitter,  lens,  and  filters.  The  lens  is  determined 
by  the  size  of  the  aperture  out  of  the  emitter,  the  size  of  the  part  to  be  illuminated,  and 
the  distance  between  the  emitter  and  the  part.  The  lens  is  specified  using  the  formula  for 
transverse  magnification  (see  equation  J.l). 


Mt 


du 


(J.l) 


where; 
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di  =  image  height 

do  =  object  height 

5,  =  image  distance  w  /  for  >/ 

/  =  focal  length  of  lens 

So  =  object  distance. 

A  diagram  of  a  lens  is  shown  in  figure  J.l.  Using  the  emitter  aperture  size  as  35  mm 


and  the  area  to  be  illuminated  as  a  circle  of  diameter  14.14"  the  resulting  transverse 
magnification  is  calculated  to  be: 


Mt  =  (14.14incfies)( 


2.54cm 

inch 


1077im 

cm 


)( 


1 

35mm 


)  =  10.26 


Using  a  lens  with  a  focal  length  of  35  mm  and  the  calculated  transverse  magnification,  the 
distance  from  the  lens  to  the  part  to  be  inspected  is  calculated  as: 


Sq 


35mm 

10.26 


=  359.16mm  =  14.147nc/ie5 


The  output  power  of  the  emitter  is  determined  by  the  desired  intensity  at  the  surface 
of  the  part,  the  transmission  of  the  filters  used,  and  the  surface  area  of  the  part  (see 
equation  J.2).  The  purpose  of  the  filter  is  to  eliminate  any  unnecessary  wavelengths  of 
light.  The  light  source  design  must  excite  the  FP,  but  not  corrupt  the  CCD  camera 
pictures.  If  the  emitter  supplies  a  light  with  a  small  bandwidth  then  this  filter  is  not 
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needed. 


Pe  = 


IqAq 

Ft 


FA, 

F 


(J.2) 


where: 

P,  =  Power  of  the  emitter  in  watts  at  490nm 
F  =  Intensity  of  light  at  the  object  surface  in  watts fm' 

Ao  =  Area  of  object  surface  in  wr 
Ft  =  Transmission  of  the  filter  at  490nm 
F  =  Intensity  of  light  at  the  aperture  of  the  emitter  in  Wjm- 
Ae  =  Area  of  aperture  of  the  emitter  in  mr. 

With  the  requirement  of  the  intensity  of  incident  light  on  the  surface  of  the  part  of 
800/iW/cm^,  the  transmission  of  the  filter  of  80%,  and  the  surface  area  to  be  illuminated 
as  156.95  square  inches  the  required  output  power  of  the  emitter  is  calculated  as: 


=  (^K 


cm‘ 


w  ^  156.59mcfie5- 2.54cm.,.  1  .  , 

iow»< — i — 


J.  3  Camera 


The  specifications  for  the  camera  depend  on  the  resolution  desired,  the  amount  of 
light  available,  and  the  size  of  the  object  to  be  imaged. 


The  resolution  desired  and  the  size  of  the  object  to  be  imaged  at  one  time  determine 
the  size  of  the  pixel  array  required  for  the  camera  (see  equation  J.3). 


P.  = 


R. 


(J.3) 


where; 

Px  =  number  of  pixels  in  the  x  direction 
F  =  length  of  imaged  object  in  x  direction 

Rx  =  resolution  in  the  x  direction 


Equation  J.3  must  be  used  for  the  vertical  and  horizontal  directions  to  calculate  the 
CCD  array  size.  In  order  to  image  an  object  10"  X  10"  with  a  resolution  of  0.005"  an 
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array  of  2000  x  2000  pixels  is  required.  Several  manufacturers  produce  a  CCD  array  size 
of  2048  X  2048  and  this  is  the  size  that  should  be  chosen  for  these  requirements. 

The  light  sensitivity  of  the  camera  is  dependent  on  the  amount  of  light  from  the 
imaged  object  and  any  ambient  light.  For  applications  using  FP,  the  amount  of  light 
emitted  from  the  FP  depends  on  the  amount  of  penetrant  on  the  object,  the  amount  of 
ultraviolet  light  incident  onto  the  FP,  and  the  efficiency  of  the  FP.  The  efficiency  of  the 
FP  is  unknown  so  trial  and  error  is  used.  Using  several  stage  nine  compressor  blades  with 
cracks  at  the  end,  a  greyscale  camera  with  a  sensitivity  of  two  lux  was  unable  to  image 
the  crack.  With  the  same  blades,  a  color  camera  with  a  sensitivity  of  0.5  lux  was  able  to 
image  the  indications. 

For  the  application  of  imaging  jet  engine  compressor  blades  10"  x  10",  a  CCD  camera 
with  a  2048  x  2048  pixel  array  and  a  sensitivity  of  0.5  lux  is  sufficient. 


J.4  Camera  Lens 

The  lens  specifications  for  the  camera  depeitd  on  the  size  of  the  array  that  the  camera 
has  for  an  image  plane,  the  size  of  the  imaged  object,  the  depth  of  the  imaged  object,  and 
the  wavelength  of  the  light  being  focused. 

The  magnification  required  is  determined  by  the  size  of  the  image  plane  and  the 
imaged  object.  Equation  J.l  is  used  to  determine  the  magnification.  As  an  example 
Photometries  KAF-4200  CCD  camera  has  an  array  of  2048  x  2048  9/xm  pixels.  To  obtain 
0.005"  resolution  an  area  of  10.24"  X  10.24"  may  be  imaged  at  one  time,  resulting  in  a 
magnification  of: 


(2048pixe/s)(9-  10-Sz)(100^)(^) 
( 2048pjxc/s )( 0.005mc/ies ) 


0.071 


The  depth  of  focus  is  determined  by  the  loss  of  center  intensity.  According  to  Bonn 
&  Wolf:  ‘‘It  is  usual  to  regard  a  loss  of  about  20  per  cent  [sic]  in  intensity  at  the  centre  [sic] 
of  the  image  patch  as  permissible”! 7).  Using  this  intensity  tolerance  the  focal  tolerance 
{AZ,)is  calculated  by  equation  J.4  (7). 
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AZ.^±i(^)=A  =  ±2(//#)-A 

2  a 

where: 

A  =  wavelength  of  light 
/  =  focal  length  of  lens 

a  =  radius  of  lens 
Z/:^  =  f-number  of  the  lens 

The  depth  of  focus  (AZo)  is  related  to  the  focal  tolerance  by  equation  J.5. 


(J.4) 


AZo  =  -(Mf-)AZi 


(J.5) 


Substituting  for  AZ,  and  multiplying  by  2  to  incorporate  the  positive  and  negative  results 
in  equation  J.6. 

AZo  =  4Mf  2(//#f  A  (J-6) 

For  the  design  specifications  of  imaging  a  blade  with  a  depth  of  4"  (AZ,,  =  4),  the  magni¬ 
fication  calculated  above,  and  the  wavelength  of  the  light  as  490nm;  the  f-number  (//#) 
is  calculated  to  be: 


//#  =  ( 


aZqMI 
4A  ’  ^ 


4(0.071)^2.54  . 

4(490-  10'®(  100)^' 


16.17 


Using  a  lens  with  a  focal  length  of  35  mm  (/  =  35mm  =  1.38")  and  imposing  equation  J.l 
results  in  an  object  distance  of; 


5o  = 


A 

Mt 


l.3Sinches 

(Ton 


l9Ainches 


J.  5  Conclusion 

The  formulas  given  here  are  sufficient  to  specify  the  sensor  system  required  for  imag¬ 
ing  jet  engine  blades  using  fluorescent  penetrant  as  long  as  the  focal  length  of  the  camera 
lens  is  much  less  than  the  distance  between  the  camera  lens  and  the  blade.  Using  these 
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equations  it  is  found  that  to  obtain  the  desired  resolution  over  a  10"  x  10"  x 
sensor  with  the  specifications  in  table  J.l  will  be  sufficient. 


array  size 

2048  X  2048 

pixel  size 

9/im  X  9/i77? 

lens  focal  length 

35mm 

lens 

16 

distance  from  lens  to  blade 

19.4  inches 

camera  sensitivity 

0.5  lux 

Table  J.l.  Sensor  Specifications 


4"  blade,  a 
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Appendix  K.  Throughput  Capacity  Calculations 


K.l  Introduction 

Throughput  capacity  is  one  of  the  five  evaluation  criteria  presented  in  chapter  V. 
The  criterion  provides  a  measure  of  each  inspection  system’s  ability  to  process  a  large 
surge  in  the  quantity  of  blades.  This  appendix  explains  the  throughput  calculations  in 
detail.  The  input  values  required  to  make  the  calculations  are  outlined  along  with  the 
computation  process  and  the  resulting  output  values.  The  calculations  for  the  system 
throughput  capacity  are  based  on  a.ssumptions  about  the  quantity  of  blades,  the  human 
inspectors,  and  the  performance  of  the  automated  inspection  system.  These  assumptions 
were  explained  fully  in  section  5.2.3  and  the  pertinent  values  are  repeated  here  for  clarity. 

K.2  Input  Values 

The  input  values  enumerated  below  were  required  to  complete  the  throughput  anal¬ 
ysis.  Some  of  these  values  were  also  used  for  the  cost  of  incorrectly  classifying  blades 
calculations.  When  a  particular  value  is  different  for  each  alternative,  the  values  are  listed 
as  a  single  vector  with  the  baseline  value  first,  followed  by  the  three  alternatives  in  order: 
[Baseline,  Alt  1,  Alt  2,  Alt  3].  For  example,  the  percentage  of  blades  sent  directly  to  the 
human  inspector  depends  on  which  alternative  being  considered,  so  the  values  are  shown 
in  a  vector  as:  [0.95,  1.0,  0.0,  0.0].  In  addition  to  the  values  used  for  the  calculations,  the 
variable  names  associated  with  the  input  values  are  also  shown. 

Inputs 

1.  Total  number  of  blades  requiring  inspection:  1.2  million,  number_blades. 

2.  Probability  of  any  blade  being  good:  0.95.  percentage_good. 

3.  Percentage  of  blades  sent  directly  to  the  human  inspector:  [0.95,  1.0,  0.0,  0.0], 
PERCENTAGE.HUMAN.DIRECTLY. 

4.  Time  required  for  a  human  inspector  to  inspect  a  good  blade:  5  seconds, 
hvunan.inspect_time_good. 
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5.  Time  required  for  a  human  inspector  to  inspect  a  bad  blade:  30  seconds. 

human. inspect _time_bad. 

6.  Time  required  for  the  automated  inspection  system  to  inspect  a  blade.  Both  good  and 
bad  blades  require  the  same  amount  of  time:  [15, 0, 10, 10]  seconds,  AUTO_INSPECT_TIME. 

7.  Probability  that  the  automated  system  sends  a  cracked  blade  to  the  human  inspector 
for  re-inspection:  [0.99,  0.0,  0.005,  0.0],  PERCENTAGE_AUT0_T0_HUMAN_BAD. 

8.  Probability  that  the  automated  system  sends  a  good  blade  to  the  human  inspector 
for  re-inspection:  [0.5,  0.0,  0.20,  0.0],  PERCENTAGE_AUT0_T0_HUMAN_G00D. 

K.3  Sample  Calculations 

The  input  values  are  transformed  into  useful  output  values  through  a  few  simple 
computations  which  are  outlined  below.  The  MathCAD  software  package  was  used  to 
complete  these  calculations.  In  the  equations,  variables  with  lowercase  letters  are  scalar 
quantities  and  variables  with  all  capital  letters  are  vectors,  as  described  for  the  input  values 
above.  When  two  vector  quantities  are  multiplied  together,  the  multiplication  is  element 
by  element  so  that  the  result  is  still  a  vector.  Note  that  some  of  the  vector  values  may  be 
^ero. 

1.  Compute  the  percentage  of  bad  blades: 

percentage.bad  =  1  —  percentage.good 

2.  Compute  the  number  of  blades  sent  directly  to  the  human  inspector: 

NUMBER.HUMAN.DIRECTLY  =  number.blades  X  PERCENTAGE.HUMAN.DIRECTLY 

3.  Compute  the  number  of  blades  sent  directly  to  the  automated  inspection  system: 

NUMBER.AUTO.DIRECTLY  =  number.blades  -  NUMBER_HUMAN_DIRECTLY 

4.  Compute  the  number  of  good  blades  and  the  number  of  bad  blades  that  the  auto¬ 
mated  inspection  system  receives: 

NUMBER.AUTO.GOOD  =  percentage.good  x  NUMBER.AUTO.DIRECTLY 

NUMBER.AUTO.BAD  =  percentage.bad  x  NUMBER.AUTO.DIRECTLY 
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5.  Compute  the  number  of  good  blades  and  the  number  of  bad  blades  that  the  auto¬ 
mated  inspection  sends  to  the  human  inspector  for  re-inspection: 

NUMBER_AUT0.T0_HUMAN_G00D  =  PERCENTAGE_AUTO_TO.HUMAN_GOOD 

X  NUMBER.AUTO.GOOD 

NUMBER_AUTO_TO_HUMAN_BAD  =  PERCENTAGE.AUTO.TO.HUMAN.BAD  X  NUMBER. AUTO _BAD 

6.  Compute  the  number  of  good  blades  and  the  number  of  bad  blades  that  the  human 
inspector  must  look  at.  This  calculation  accounts  for  blades  sent  directly  to  the 
human  inspector  and  those  received  from  the  automated  system: 

NUMBER.HUMAN.GCOD  =  percentage.good  X  NUMBER.HUMAN.DIRECTLY 

+  NUMBER.AUTO.TO.HUMAN.GOOD 

NUMBER.HUMAN.BAD  =  percentage.bad  X  NUMBER.HUMAN.DIRECTLY 

-I-  NUMBER. AUTO  .TO.HUMAN.BAD 

7.  Compute  the  time  (in  seconds)  required  for  the  automated  inspection  system  to 
process  the  blades: 


TIME.AUTO  =  AUTO.INSPECT.TIME  X  NUMBER.AUTO.DIRECTLY 

8.  Compute  the  time  (in  seconds)  required  for  the  human  inspector  to  process  the 
blades: 

TIME.HUMAN  =  human.inspect.time.good  X  NUMBER.HUMAN.GOOD 
-I-  human_inspect.time.bad  x  NUMBER.HUMAN.BAD 

9.  Compute  the  total  inspection  time: 


TOTAL.TIME.SEC  =  TIME.HUMAN  -f  TIME.AUTO 

10.  Convert  all  inspection  times  from  units  of  seconds  to  units  of  hours: 

TOTAL.TIME.HOURS  = 

11.  Compute  the  average  number  of  blades  inspected  per  hour: 


BLADES.PER.HOUR 


number -blades _ 

TOTAL-TIME-HOUFIS 
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K.4  Output  Values 


Table  K.l  shows  the  results  of  applying  the  above  computations  to  the  input  values. 
Only  two  of  the  types  of  values  reported  in  table  K.l  are  significant  outputs  that  infiuence 
the  results  of  the  study. 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Automated  hours  of  operation 

250 

0 

3333 

3333 

Manual  hours  of  inspection 

2044 

2083 

319 

0 

Total  hours  per  year 

2294 

2083 

3652 

3333 

Average  blades  per  hour 

523 

576 

329 

360 

Table  K.l.  Summary  of  throughput  calculations. 


Outputs 


1.  The  total  number  of  inspection  hours  required  for  the  human  inspectors. 

2.  The  average  number  of  blades  inspected  per  hour. 

The  first  output  value,  the  number  of  inspection  hours,  was  used  for  the  equivalent 
uniform  annual  cost  analysis  to  determine  the  number  of  labor  hours  required  for  inspec¬ 
tion.  The  other  output  value,  the  average  number  of  blades  inspected  per  hour,  was  used 
to  measure  the  throughput  capacity  of  each  alternative,  as  discussed  in  section  5.6.  Recall 
that  the  throughput  capacity  criterion  provides  a  measure  of  each  alternative’s  ability  to 
handle  large  surges  in  the  number  of  blades  requiring  inspection.  In  addition,  the  num¬ 
ber  of  blades  inspected  per  hour  was  also  a  useful  input  for  the  upgradability  analysis, 
discussed  in  section  5.7.  In  terms  of  upgradability,  an  increase  in  throughput  capacity 
provides  more  flexibility  in  how  the  system  can  be  operated  to  handle  a  large  volume  of 
blades. 

K.5  Summary 

Throughput  capacity  was  measured  in  terms  of  the  average  number  of  blades  per 
hour  that  each  inspection  system  could  process.  The  specific  computations  required  to 
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determine  the  throughput  surge  capacity  of  each  alternative  were  explained  in  detail.  The 
necessary  inputs  for  the  calculations  and  the  important  outputs  from  the  analysis  and  their 
relationship  to  other  evaluation  criteria  were  also  discussed. 
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Appendix  L.  Sensitiviiy  Analysis  of  Sysl(  m  Evahiafion 


L.l  Introduction 

The  results  of  the  evaluation  presented  in  chapter  V  depend  heavily  on  the  weights 
assigned  to  the  five  criteria.  The  weights  were  developed  using  a  preference  chart  and 
following  the  procedure  outlined  by  Athey  (2).  As  mentioned  in  chapter  V,  the  weights 
in  the  chart  could  be  adjusted  to  reflect  a  different  perspective  on  the  relative  importance 
of  the  criteria.  This  appendix  shows  one  example  of  a  different  preference  chart,  how  it 
would  change  the  relative  weights,  and  how  the  new  weights  influence  the  final  evaluation 
of  the  alternatives. 

L.2  An  Alternative  Preference  Chart 

The  purpose  of  the  preference  chart  is  to  establish  the  rank  order  of  the  important 
criteria  involved  in  the  evaluation  and  to  determine  the  weights  that  should  be  associated 
with  each  criterion.  As  explained  in  chapter  V,  the  chart  uses  simple  pair-wise  compar¬ 
isons  between  all  criteria  to  calculate  the  appropriate  weiglits.  The  symbols  used  in  the 
preference  chart,  their  meaning,  and  point  values  are  summarized  in  table  L.l. 


Symbol 

> 

4 

> 

more 

3 

= 

equal 

2 

< 

less 

1 

< 

much  less 

0 

Table  L.l.  Comparison  symbols  for  the  preference  chart. 


The  preference  chart  developed  for  the  original  evaluation  considered  the  cost  of 
incorrectly  classifying  blades  (misclassification)  as  the  dominant  criterion.  Correctly  clas¬ 
sifying  blades  is  the  paramount  mission  of  the  IBIS  facility,  so  it  is  reasonable  that  the 
evaluation  depends  heavily  on  the  system's  ability  to  accurately  inspect  blades.  The  mis¬ 
classification  criterion  received  a  weight  that  was  approximately  twice  as  large  as  the  weight 
for  the  next  closest  criterion,  maintainability. 
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The  original  evaluation  process  attempted  to  approach  the  problem  of  blade  inspec¬ 
tion  from  the  point  of  view  of  what  is  best  for  the  Air  Force  overall.  Suppose  a  slightly 
more  myopic  view  of  the  inspection  process  was  developed,  such  that  the  maintainability 
of  the  inspection  system  was  considered  the  most  important  criterion.  A  person  taking 
this  stance  would  want  a  maintainable  system  above  anything  else.  The  quality  of  the 
blade  inspection  process  would  be  secondary  to  maintainability.  The  order  of  the  three 
remaining  alternatives  would  be  unchanged  from  the  original  evaluation.  The  preference 
chart  which  reflects  the  new  perspective  is  shown  in  table  L.2. 


Criteria 

Cost 

Thru 

U  pgrade 

Points 

Weight 

Misclassification 

< 

> 

> 

> 

9 

22.. a 

Maintainability 

> 

> 

> 

» 

40.0 

Annual  Cost 

< 

< 

> 

> 

17.0 

Throughput 

< 

< 

< 

> 

5 

12. .a 

Upgradability 

< 

< 

< 

< 

Table  L.2.  Alternative  preference  chart.  The  chart  reflects  the  stance  that  maintainabil¬ 
ity  is  the  most  important  of  the  five  criteria. 

The  new  preference  chart  illustrates  that  several  changes  would  have  to  be  made  to 
the  original  preference  chart  in  order  to  exchange  the  weights  of  the  two  most  important 
criteria.  In  the  new  chart,  all  of  the  symbols  associated  with  the  misclassification  row 
and  the  maintainability  row  w'ere  changed  to  reflect  the  new  perspective  on  the  problem. 
For  this  example,  maintainability  is  now  the  dominant  criterion,  with  misclassification  a 
distant  second.  Similar  changes  could  be  made  to  increase  the  importance  of  any  criterion, 
depending  on  how  the  person  completing  the  chart  views  tlie  inspection  problem  and  the 
individual  criterion. 

It  is  important  to  note  that  the  new  chart  only  influences  the  relative  weights  in  the 
evaluation  matrix.  (Recall  that  the  weights  were  shown  in  parentheses  at  the  left  liand 
side  of  the  evaluation  matrix  in  table  .").2o.)  The  utility  values  and  confidence  factors 
established  in  chapter  V  were  developed  separately  and  are  completely  independent  from 
the  relative  weights.  The  independence  between  the  weights  and  the  other  two  values  makes 
it  relatively  easy  to  adjust  the  particular  nre.'erenre  a[)plied  to  the  evaluation  proce.ss 
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without  requiring  a  complete  re-evaluation  of  all  of  the  ciiieria.  In  brief,  changing  the 
weights  in  the  preference  chart  does  not  change  any  of  the  utility  values  or  confidence 
factors  in  the  evaluation  matrix. 

L.3  New  Evaluation  of  Alternatives 

Even  though  the  weights  developed  using  the  new  preference  chart  are  independent 
of  the  utility  scores  and  confidence  factors,  the  three  values  interact  when  completing  the 
evaluation  matrix.  The  interaction  causes  the  final  evaluation  scores  for  the  alternatives 
to  change.  The  critical  numbers  for  this  example  are  the  total  values  and  the  discounted 
values,  which  are  displayed  in  the  bottom  section  of  the  evaluation  matrix.  Table  L.3 
summarizes  the  results  if  the  new  weights  are  applied  to  the  alternatives  without  changing 
the  utility  or  confidence  values  in  the  original  evaluation  matrix.  (See  table  5.25  for  the 
original  matrix.) 


Baseline 

Alt  1 

Alt  2 

Alt  3 

Total  Value  (U) 

^  437.3 

561.8 

560.0 

563.5 

Total  Discounted  Value  (D) 

299.9 

366.0 

278.3 

229.7 

Table  L.3.  Results  of  the  evaluation  with  new  criteria  weights. 


The  new  scores  reveal  that  there  is  only  a  slight  change  in  the  final  result  in  terms  of 
the  total  value  of  the  alternatives.  Alternative  3  still  has  the  highest  total  value,  followed 
by  alternative  1  and  alternative  2.  The  baseline  system  still  has  the  lowest  score  for  the 
total  value.  Note  that  the  total  value  scores  for  alternatives  1  and  2  are  very  close  to  the 
score  for  alternative  3.  The  difference  is  only  0.3%  between  alternative  1  and  alternative  3 
ana  0.6%  between  alternative  2  and  alternative  3.  These  minor  differences  indicate  that 
the  three  alternatives  are  essentially  equivalent  in  terms  of  the  total  value  score,  while  the 
baseline  system  is  significantly  worse  (22%  lower  than  alternative  1). 

In  the  original  evaluation,  alternative  3  had  the  highest  total  value  score  and  was 
followed  by  alternative  2.  Even  after  making  significant  changes  to  the  preference  chart, 
alternatives  2  and  3  still  received  high  total  value  scores.  Since  alternative  3  remained  at 
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the  top  in  both  evaluations,  it  is  reasonable  to  conclude  that  the  original  evaluation  is  not 
affected  by  minor  changes  in  the  evaluation  matrix.  In  other  words,  the  final  scores  for  tlie 
cilternatives  in  terms  of  total  value  are  not  sensitive  to  the  slight  changes  in  the  preference 
chart. 

Compared  to  the  original  evaluation,  the  order  of  the  alternatives  also  changed  when 
considering  the  total  discounted  values  shown  in  table  L.'.i.  .Alternative  1  still  received 
the  highest  discounted  utility,  so  the  recommendation  about  what  design  to  pursue  would 
not  change.  The  order  of  the  remaining  alternatives  and  the  baseline  changed  slightly. 
Alternative  1  was  followed  in  order  by  the  baseline  system,  alternative  2,  and  alternative  3. 
As  with  the  total  value  scores,  the  total  discounted  values  are  not  very  sensitive  to  minor 
changes  in  the  preference  chart. 

L.4  Summary 

This  analysis  considered  the  sensitivity  of  the  evaluation  process  to  changes  in  the 
preference  chart.  Tlie  preference  chart  was  used  to  develop  weights  that  measure  the  rel¬ 
ative  importance  of  each  criterion.  For  this  example,  the  stance  that  the  maintainability 
criterion  was  more  important  than  the  cost  of  incorrectly  classifying  blades  was  applied 
to  the  preference  chart  to  develop  a  new  set  of  weights.  The  new  weights  were  incorpo¬ 
rated  into  the  original  evaluation  matrix  and  the  resulting  scores  for  total  value  and  total 
discounted  value  were  compared  to  the  original  results.  The  comparison  revealed  that  the 
evaluation  process  was  not  particularly  sensitive  to  minor  perturbations  in  the  weights  as¬ 
sociate  with  each  criterion.  Small  changes  could  be  made  to  the  preference  chart  without 
affecting  the  final  results  of  the  evaluation  process. 

There  are  two  important  observations  associated  with  this  sensitivity  analysis.  First, 
it  is  relatively  easy  to  adjust  the  weights  associated  with  the  criteria  to  reflect  a  different 
perspective  on  the  blade  inspection  problem.  Developing  the  preference  chart  involves 
some  subjective  judgement,  so  it  is  reasonable  to  assume  that  different  people  will  place 
different  levels  of  importance  on  the  evaluation  criteria.  Since  the  weights  can  be  adjusted 
easily,  it  is  likewise  a  simple  process  to  determine  the  effect  a  different  perspective  has  on 
the  final  evaluation  of  the  alternatives.  The  second  important  point  is  that  the  weights 
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developed  with  the  preference  chart  are  completely  independent  of  the  utility  scores  and 
confidence  factors  assigned  to  each  alternative  in  chapter  \'.  Changing  the  weights  for  the 
criteria  will  not  cause  a  change  in  the  individual  utility  scores  or  confidence  factors,  but 
may  cause  a  change  in  the  final  evaluation  because  the  three  values  interact  within  the 
evaluation  matrix. 

Finally,  this  example  is  for  illustrative  purposes  only.  It  demonstrates  the  robustness 
of  the  original  evaluation,  but  does  not  in  any  way  alter  the  final  decision  or  recommen¬ 
dations  discussed  in  chapter  VI. 


Appendix  M.  Decision  Analysis  Based  on  Cost 


The  decision  analysis  performed  in  chapter  V  is  based  on  Athey's  Systematic  Systems 
Approach.  This  systems  engineering  method  calls  for  the  establishment  of  criteria  upon 
which  feasible  alternatives  can  be  evaluated.  The  criteria  are  weighted  in  terms  of  relative 
importance.  The  feasible  alternatives  receive  a  utility  score  in  each  of  the  criteria,  and 
these  scores  are  multiplied  by  the  criteria  weighting  factor  and  summed  for  each  criterion. 
The  sum  represents  the  total  utility  based  the  alternatives  ability  to  address  the  criteria. 
This  approach  identifies  the  alternative  with  the  highest  utility  as  best.  A  cost  analysis 
represents  a  second  approach  to  the  selection  of  the  best  alternative. 

The  performance  of  a  cost  analysis  relies  on  good  estimates  of  the  cost  implications 
for  the  various  alternatives.  Cost  estimates  for  this  analysis  exist  in  the  decision  analysis 
of  chapter  V.  These  estimates  emerge  ajs  the  cost  of  incorrectly  classifying  a  blade  (CICB) 
and  the  equivalent  uniform  annual  cost  (EUAC).  These  costs  are  a  direct  reflection  of  the 
same  two  decision  criteria  used  in  chapter  V.  The  EUAC  also  indirectly  represents  th  ' 
maintainability  and  the  throughput  capacity  criteria  since  these  two  criteria  contribute 
directly  to  EUAC.  The  fifth  criterion,  upgradability,  does  not  translate  to  cost  terms  and 
therefore,  is  not  reflected  in  the  cost  analysis. 

Table  M.l  presents  the  CICB  and  EUAC  final  values  from  chapter  V  as  well  as  a  cost 
avoidance  amount.  The  cost  avoidance  is  the  amount  of  money  saved  by  the  Air  Force  due 
to  the  implementation  of  one  of  the  three  alternatives.  The  cost  avoidance  is  calculated 
by  subtracting  the  total  cost  associated  with  each  of  the  three  alternatives  from  the  total 
cost  associated  with  the  baseline  system. 

The  fact  that  Alternative  1  has  a  higher  cost  than  the  baseline  emerges  as  a  trouble¬ 
some  aspect  of  the  cost  avoidance  analysis.  Since  the  SQC  plan  is  a  primary  aspect  of  the 
first  alternative,  the  financial  benefits  of  the  SQC  plan  should  be  clarified.  An  accurate 
measure  of  the  benefits  of  the  SQC  plan  in  terms  of  dollars  is  infeasible.  As  a  result  the  cost 
benefits  of  the  SQC  plan  are  not  included  in  this  desision  analysis  based  on  cost,  however 
appendix  B  presents  a  break  even  analysis  and  an  SQC  cost  avoidance  versus  inspection 
characteristic  curve. 
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CICB($) 

EUAC($) 

Total  Cost($) 

Cost  .4voidance(S) 

Baseline 

94.90  M 

1.238  M 

96.14  M 

0 

Alt  1 

95.03  M 

1.192  M 

96.22  M 

-0.08  M 

Alt  2 

36.08  M 

1.351  M 

37.43  M 

58.71  M 

Alt  3 

24.46  M 

1.442  M 

25.90  M 

70.24  M 

Table  M.l.  Cost  Analysis 


The  cost  avoidance  dollar  amount  represents  a  true  savings  to  the  Air  Force  if  the 
estimates  are  valid.  As  mentioned  in  appendix  D  the  model  used  for  the  CICB  calculations 
overestimate  the  number  of  bad  units  assembled,  and  hence  the  estimated  costs  are  also 
over  inflated.  Assuming  the  CICB  values  are  an  order  of  magnitude  high,  even  though  the 
model  validation  indicates  they  are  closer  than  this,  a  second  cost  analysis  was  performed. 
Table  M.2  presents  the  values  for  the  second  analysis.  A  significant  annual  cost  avoidance 
results  from  both  the  analyses  with  the  calculated  CICB  values  and  the  reduced  CICB 
values.  The  Air  Force  will  realize  the  largest  savings  through  the  implementation  of  a  fully 
automated  inspection  system. 


Reduced  CICB($) 

EUAC($) 

Total  Cost($) 

Cost  Avoidance($) 

Baseline 

9.490  M 

1.238  M 

10.728  M 

0 

Alt  1 

9.503  M 

1.192  M 

10.695  M 

0.033  M 

Alt  2 

3.608  M 

1.351  M 

4.959  M 

5.769  M 

Alt  3 

2.446  M 

1.442  M  i 

3.888  M 

6.840  M 

Table  M.2.  Second  Cost  Analysis 


In  order  to  approach  the  cost  analysis  of  IBIS  upgrades  from  a  different  perspective, 
the  question  to  address  is,  at  what  point  does  an  investment  to  improve  the  IBIS  begin  to 
show  a  return  for  the  Air  Force?  A  comparison  of  the  unscheduled  maintenance  event  cost 
of  $58,500  to  the  difference  of  the  baseline  EU.\C  and  the  alternative  3  EUAC  will  derive 
a  break  even  value  for  such  an  investment. 

(Alt  3  EUAC  -  Baseline  EUAC)  =  ($1,442  M  -  $1,238  M)  =  $0,204  M 
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A  division  of  $204,000  by  $58,500  yields  3.5.  Therefore,  if  such  an  improved  system 
prevents  4  additional  critically  cracked  blades  from  entering  service,  thus  preventing  4 
unscheduled  maintenance  events,  the  Air  Force  will  realize  a  gain  on  its  investment.  This 
break  even  point  is  appropriate  if  the  EUAC  estimates  are  accurate  and  if  each  missed 
detection  causes  a  maintenance  event.  Table  M.3  addresses  this  concern  by  presenting 
the  additional  number  of  critical  cracks  that  must  be  found  over  a  1  year  period  for  the 
improved  system  to  break  even.  Alternative  3  is  expected  to  prevent  additional  critically 
cracked  blades  from  entering  service  in  e.xcess  of  the  values  presented  in  table  M.3. 


Annual 

Investment($) 

Additional  Cracks 
Found  (blades) 

204,000 

4 

250,000 

5 

500.000 

9 

750,000 

13 

995,000 

17 

Table  M.3.  Cost  Analysis.  Break  Even  Point 

The  basic  tenet  of  this  cost  avoidance  presentation  is  that  an  improvement  to  the 
baseline  system  should  not  be  judged  solely  on  the  local  impact  at  the  SA-ALC.  The  Air 
Force  will  realize  a  return  on  the  investment  made  by  the  SA-ALC  to  improve  the  IBIS.  If 
an  improved  inspection  system  will  correctly  classify  additional  critical  cracks  in  jet  engine 
compressor  and  turbine  blades,  unscheduled  maintenance  events  will  be  avoided.  Each 
unscheduled  maintenance  event  avoided  saves  the  Air  Force  $58,500.  It  is  important  to 
note  the  this  cost  analysis  does  not  address  the  impact  of  a  catastrophic  failure  due  to 
poorly  inspected  compressor  and  turbine  blades.  Such  an  impact  is  difficult  to  quantify, 
but  any  improvements  to  the  IBIS  that  will  increase  the  number  of  critically  cracked  blades 
removed  from  service  will  reduce  the  likelihood  of  a  catastrophic  failure. 

This  cost  analysis  of  the  three  alternatives  of  the  IBIS  upgrade  study  calls  for  the 
implementation  of  the  fully  automated  inspection  system.  This  result  agrees  with  the 
result  of  the  decision  analysis  performed  in  chapter  V.  The  fully  automated  inspection 
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system  will  correctly  classify  more  blades  to  the  extent  that  the  Air  Force  will  save  money 
over  the  current  operation  of  the  IBIS  as  reflected  by  the  baseline  system. 


Appendix  N.  List  of  Acronyms 


ABR 

Automated  Blade  Reader 

AFIT 

Air  Force  Institute  of  Technology 

AFPPM 

Automated  Fluorescent  Penetrant  Preprocessing  Module 

CAD 

Computer  Aided  Design 

CCD 

Charge  Coupled  Device 

CICB 

Cost  of  Incorrectly  Classifying  a  Blade 

CONUS 

Continental  United  States 

DCP 

Durability  Critical  Parts 

DEC 

Digital  Equipment  Corporation 

DOE 

Design  of  Experiments 

DRAM 

Dynamic  Random  Access  Memory 

ENSIP 

Engine  Structural  Integrity  Program 

EUAC 

Equivalent  Uniform  Annual  Cost 

FCP 

Fracture  Critical  Parts 

FMECA 

Failure  Mode  Effects  Criticality  Analysis 

FOD 

Foreign  Object  Damage 

FP 

Fluorescent  Penetrant 

FPI 

Fluorescent  Penetrant  Inspection 

FPIM 

Fluorescent  Penetrant  Inspection  Module 

GE 

General  Electric 

GPIB 

General  Purpose  Interface  Bus 

lABR 

Intelligent  Automated  Blade  Reader 

IBIS 

Integrate  Blade  Inspection  System 

ICS 

Information  Computer  System 

IPGS 

Inspection  Plan  Generation  System 

IR 

Infrared 

IRIM 

Infrared  Inspection  Module 
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KNN 

k- Nearest  Neighbors 

LCC 

Life  Cycle  Cost 

MAP 

Manufacturing  Automation  Protocol 

MLP 

Multilayer  Perceptron 

MTBF 

Mean  Time  Between  Failure 

MTTR 

Mean  Time  To  Repair 

NDE 

Non-Destructive  Evaluation 

NDI 

Non- Destructive  Inspection 

NN 

Neural  Network 

NPV 

Net  Present  Value 

PC 

Personal  Computer 

PMT 

Photomultiplier  Tube 

POD(a) 

Probability  of  Detection  for  a  crack  of  length  a 

RACE 

Robotics  and  Automation  Center  of  Excellence 

RBF 

Radial  Basis  Function 

RT 

Rupture  Time 

SA-ALC 

San  Antonio  Air  Logistics  Center 

SCARA 

Selective  Compliance  Assembly  Robot  Arm 

SCSI 

Small  Computer  System  Interface 

SQC 

Statistical  Quality  Control 

TO 

Technical  Order 

UPP 

Unified  Program  Plan 

uv 

Ultraviolet 

VIM 

Visual  Inspection  Module 
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Upon  graduation  from  Basic  Military  Training  School,  John  attended  the  USAF  Precision 
Measuring  Equipment  Laboratory  (PMEL)  School  at  Lowery  AFB  in  Colorado.  Following 
PMEL  school,  John  was  assigned  to  the  Avionics  Laboratory  at  Wright- Patterson  AFB. 
Ohio  as  an  electronic  technician.  In  1985,  he  was  selected  for  the  Airman  Education  and 
Commissioning  Program  (AECP)  and  attended  Wright  State  University  where  he  received 
his  Bachelor  of  Science  degree  in  December  of  1987.  After  graduation,  he  was  commis¬ 
sioned  through  the  USAF  Officer  Training  School  and  assigned  to  the  Weapons  Laboratory, 
Kirtland  AFB,  New  Mexico.  During  his  assignment  to  the  Weapons  Laboratory,  he  per¬ 
formed  the  duties  of  Space  Systems  Engineer.  In  May  of  1991  John  entered  the  Air  Force 
Institute  of  Technology,  School  of  Engineering.  Captain  Snyder  is  married  to  the  former 
Mary  LeBarre  and  they  have  two  daughters,  Jessica  and  Cassandra. 
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Lieutenant  Gregory  J.  Toussaint  was  born  on  12  October  1967.  in  St.  Johnsbury. 
Vermont.  He  graduated  from  high  school  in  West  Hartford,  Connecticut  in  1985,  and 
received  a  Bachelor  of  Science  degree  in  Electrical  Engineering  from  Cornell  University 
in  1989.  Upon  graduation,  he  was  commissioned  through  the  Reserve  Officer  Training 
Corps  program  and  assigned  to  the  Avionics  Directorate  of  the  Wright  Laboratory,  Wright- 
Patterson  AFB,  Ohio.  In  the  laboratory,  he  performed  research  as  a  laser  communications 
engineer  until  entering  the  Air  Force  Institute  of  Technology,  School  of  Engineering,  in 
May  1991. 
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The  purpose  of  this  design  study  was  to  identify  ways  to  improve  the  Integrated  Blade  Inspection  System.  The  Air 
Force  requires  inspection  of  jet  engine  compressor  and  turbine  blades  to  locate  defects  and  prevent  engine  failure. 
The  current  inspection  process  uses  fluorescent  penetrant  as  an  aid  lo  identify  cracked  blades.  A  systems  engineer¬ 
ing  design  process  was  applied  to  evaluate  the  current  inspection  techniques  and  to  develop  alternative  methods  to 
satisfy  the  Air  Force  requirements.  Three  different  inspection  systems  were  developed  and  compared  to  the  current 
process:  manual,  semi- automated,  and  fully  automated  inspection.  This  study  made  several  noteworthy  contribu¬ 
tions:  development  of  classification  software  to  validate  the  neural  network  approach  for  accurate  blade  classification, 
demonstration  of  potential  advantages  of  charge-coupled  device  cameras  for  data  gathering,  quantification  of  the  cost 
of  incorrectly  classifying  jet  engine  blades,  examination  of  the  value  of  a  statistical  quality  control  plan  for  the  inspec¬ 
tion  process,  and  identification  of  a  method  using  multiple  images  to  extract  additional  features  from  cracks.  The 
study  demonstrates  that  the  fully  automated  system  could  dramatically  outperform  the  manual  inspection  process 
by  improving  the  consistency  of  the  inspection  process  and  raising  the  quality  of  the  blades  returned  to  service. 
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